--- title: 'Prompt Caching' source: 'https://academia.sh/en/courses/model-apis/prompt-caching' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Distinguishes the prefix cache, keyed by content, from the response cache, keyed by question, and writes both with the standard library.' - 'Measures on the same workload how hits, tokens and the stale-response share change as the cache lifetime grows.' - 'Ties the stale-response share to the drop in the correct-response share and tests the loss against the run-to-run difference.' - "Counts how the two caches change each other's hit rate when they are used together." --- # Prompt Caching Measuring the same 240 fictional requests under seven cache setups: 175 of the prompt's 186 tokens are shared by every request. The prefix cache, keyed by content, cuts input tokens from 44367 to 7472 with a 0.9792 hit rate and leaves the correct-response share unchanged. In the response cache, keyed by question, the hit rate rises from 0.4708 to 0.8792 as the lifetime grows and calls fall from 240 to 29, but the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. Together the two caches cut input to 2946, fifteenfold, and the prefix hit rate falls from 0.9792 to 0.9359 once the response cache is on. In the previous lesson the heavy load made 295 calls, and every one of them began with the same instruction. Retry multiplied that repetition once more: every call sent as a correction tokenized the shared text once again and paid for it once again. The repetition itself had not been measured. In a support assistant this repetition has two layers. The first is the **prefix of the prompt**: the instruction and the field team's fictional document pool are the same in every request and take up most of the prompt. The second is **the question itself**: a few questions produce half of all traffic, and the same question is asked dozens of times a day. Each repetition has a remedy, and **the two remedies are not the same thing**. This lesson builds the two separately and measures them side by side: as the cache hit rate rises, tokens fall, but a stored response acquires an **age**, and when the document it rests on changes, the response is still fast, still cheap and now wrong. - **PQ9.** The code is a **simulator**; no real endpoint is called. The pool, the questions and the update schedule are **fictional**, the seed is **20260218**. - **PQ10.** The pool is **ten fictional documents**; each document has a **version**, and two documents are updated at each of four separate steps. - **PQ11.** The prompt has two parts: the **prefix**, identical in every request (the instruction and the pool), and the question itself. - **PQ12. The prefix cache is keyed by content.** If the prefix is identical to the previous call's, it counts as a hit and **ten percent** of the prefix tokens are paid. If the pool changes, the prefix changes too. - **PQ13. The response cache is keyed by question** and has a **lifetime**. A hit within the lifetime makes no call and spends no tokens. - **PQ14. A stale response** is a response that comes back from the cache and whose underlying document's version is no longer current. - **PQ15. A correct response** is a response whose label is correct **and** that is not stale; this is the product measure of this lesson. - **PQ16.** The workload is 240 requests over 120 steps, two per step; the question distribution is skewed. ## Two Separate Caches, Two Separate Keys First the pool and the questions are built, then one document is updated and the answer to the same question is checked. The distinction becomes visible here: the prefix changes **by itself** when the pool changes; a stored response does not. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called. The document pool, the questions # and the update schedule are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF STEPS, PREFIX_DISCOUNT = 120, 0.10 # run length, share of a hit prefix that is paid def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] DOCUMENTS = [{"code": "FT1", "topic": "fatura", "keyphrase": "fatura tutari", "text": "fatura tutari hesap donemi ve gecikme farki kurallari"}, {"code": "FT2", "topic": "fatura", "keyphrase": "fatura odemesi", "text": "fatura odemesi yontemleri ve borc gorunumu kurallari"}, {"code": "AR1", "topic": "ariza", "keyphrase": "ariza kaydi", "text": "ariza kaydi acma ve ekip yonlendirme yordami"}, {"code": "AR2", "topic": "ariza", "keyphrase": "baglanti bozuklugu", "text": "baglanti bozuk oldugunda uygulanan denetim adimlari"}, {"code": "SY1", "topic": "sayac", "keyphrase": "sayac endeksi", "text": "sayac endeks okumasi ve duzeltme yordami"}, {"code": "SY2", "topic": "sayac", "keyphrase": "sayac degisimi", "text": "sayac degisimi randevu kurallari ve sure taahhudu"}, {"code": "TA1", "topic": "tarife", "keyphrase": "tarife basamagi", "text": "tarife basamaklari ve abone tipine gore birim ucret"}, {"code": "TA2", "topic": "tarife", "keyphrase": "tarife degisikligi", "text": "tarife degisikligi basvurusu ve gecerlilik kosullari"}, {"code": "KS1", "topic": "kesinti", "keyphrase": "planli kesinti", "text": "planli kesinti duyurusu yayimlama ve takip kurallari"}, {"code": "KS2", "topic": "kesinti", "keyphrase": "mahalle kesintisi", "text": "mahallede kesinti bildirimi ve tahmini donus suresi"}] for b in DOCUMENTS: b["version"] = 1 QUESTIONS = [{"doc": i, "topic": b["topic"], "text": tpl.format(a=b["keyphrase"])} for i, b in enumerate(DOCUMENTS) for tpl in ("{a} konusunda ne yapmam gerekiyor", "{a} icin gecerli kural nedir", "{a} ile ilgili son durum ne")] WEIGHTS = [1.0 / (i + 1) ** 1.0 for i in range(len(QUESTIONS))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] UPDATES = {24: [6, 1], 48: [0, 8], 72: [2, 6], 96: [4, 7]} # step -> document CUES = {"fatur": "fatura", "tutar": "fatura", "borc": "fatura", "odeme": "fatura", "ariza": "ariza", "bozuk": "ariza", "bagla": "ariza", "kayit": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} INSTRUCTION = ("Asagidaki dagarcigi kullanarak soruyu yanitla . Yanitta once etiket , " "sonra dayandigin belge kodu ve surumu yaz . Dagarcik: ") COUNTER = [0] def prefix(): return INSTRUCTION + " ; ".join(f"{b['code']} s{b['version']} {b['text']}" for b in DOCUMENTS) + " . Soru: " def call(question, seed=SEED): """FICTIONAL endpoint. Reads the current version of the pool.""" COUNTER[0] += 1 prompt = prefix() + question["text"] tok = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in tokenize(question["text"]): if CUES.get(t) in p: p[CUES[t]] += 1.0 r = rng((seed ^ trace_id(question["text"])) + 7919 * COUNTER[0]) e = max(sorted(p), key=lambda a: p[a]) if r() > 0.10 else TOPICS[int(r() * 5)] b = DOCUMENTS[question["doc"]] return {"label": e, "code": b["code"], "version": b["version"], "input": len(tok), "prefix": len(tokenize(prefix())), "output": len(tokenize(f"{e} {b['code']} s{b['version']}"))} def workload(seed=SEED): """Two requests per step; the question distribution is skewed (a few dominate).""" r = rng(seed + 991) return [(a, QUESTIONS[pick(r(), WEIGHTS)]) for a in range(STEPS) for _ in range(2)] for b in DOCUMENTS: b["version"] = 1 first = call(QUESTIONS[0]) print("first call :", {k: first[k] for k in ("label", "code", "version", "input", "prefix")}) DOCUMENTS[0]["version"] += 1 # pool updated second = call(QUESTIONS[0]) print("after the update:", {k: second[k] for k in ("code", "version", "input")}) print("stored response :", first["code"], "s" + str(first["version"]), "-> stale" if first["version"] != DOCUMENTS[0]["version"] else "-> fresh") DOCUMENTS[0]["version"] = 1 reqs = workload() freq = {} for _, q in reqs: freq[q["text"][:26]] = freq.get(q["text"][:26], 0) + 1 print(f"\nrequests {len(reqs)} , unique questions {len(freq)} , prefix {first['prefix']} tokens") print("top three questions:", [(a, n) for a, n in sorted(freq.items(), key=lambda x: -x[1])[:3]]) ``` ``` first call : {'label': 'fatura', 'code': 'FT1', 'version': 1, 'input': 186, 'prefix': 175} after the update: {'code': 'FT1', 'version': 2, 'input': 186} stored response : FT1 s1 -> stale requests 240 , unique questions 29 , prefix 175 tokens top three questions: [('fatura tutari konusunda ne', 56), ('fatura tutari icin gecerli', 41), ('fatura tutari ile ilgili s', 23)] ``` The first line gives the size of the repetition: the prompt is 186 tokens, and **175 of them are prefix**. That is, **0.9409** of the tokens each request pays for are identical to the previous request's. The second line shows that the call after the update reads the same document in its second version. The third line is the distinction itself: the stored response still rests on the first version and has become **stale**. The question distribution matters too. The 240 requests correspond to only 29 distinct questions, and the three most frequent questions alone account for 120 requests. A response cache works only because of this skew; on a workload where every request is different, the response cache's hit rate stays at zero. ## Stale-Response Share as the Hit Rate Rises Seven setups are run on the same workload: the no-cache baseline, the prefix cache only, the response cache with four different lifetimes, and both caches together. ```python def run(prefix_on, lifetime=0, seed=SEED): """prefix_on: is the prefix cache on . lifetime: step lifetime of the response cache.""" for b in DOCUMENTS: b["version"] = 1 COUNTER[0] = 0 cache, last, prev_step = {}, [None], [-1] s = dict.fromkeys(("prefix_hits", "resp_hits", "input", "output", "stale", "correct", "requests"), 0) for step, question in workload(seed): if step != prev_step[0]: prev_step[0] = step for i in UPDATES.get(step, []): DOCUMENTS[i]["version"] += 1 s["requests"] += 1 entry = cache.get(question["text"]) if lifetime and entry and step - entry["step"] <= lifetime: s["resp_hits"] += 1 stale = entry["version"] != DOCUMENTS[question["doc"]]["version"] s["stale"] += stale s["correct"] += entry["label"] == question["topic"] and not stale continue resp = call(question, seed) s["output"] += resp["output"] if prefix_on and last[0] == prefix(): # prefix is keyed by content s["prefix_hits"] += 1 s["input"] += (round(resp["prefix"] * PREFIX_DISCOUNT) + resp["input"] - resp["prefix"]) else: s["input"] += resp["input"] last[0] = prefix() cache[question["text"]] = {"label": resp["label"], "version": resp["version"], "step": step} s["correct"] += resp["label"] == question["topic"] n, c = s["requests"], max(COUNTER[0], 1) return (s["resp_hits"] / n, s["prefix_hits"] / c, s["input"], s["input"] + s["output"], COUNTER[0], s["stale"] / n, s["correct"] / n) SETUPS = [("no cache", 0, 0), ("prefix cache", 1, 0), ("response lifetime 8", 0, 8), ("response lifetime 24", 0, 24), ("response lifetime 64", 0, 64), ("response lifetime unlimited", 0, 10 ** 6), ("prefix + response lifetime 24", 1, 24)] print("setup resp hit prefix hit input total" " calls stale correct") for name, o, m in SETUPS: a = run(o, m) print(f"{name:<29} {a[0]:>8.4f} {a[1]:>10.4f} {a[2]:>6} {a[3]:>6} {a[4]:>5}" f" {a[5]:>6.4f} {a[6]:>7.4f}") print("\nsecond run (seed 20260219), response hit / stale / correct:") for name, o, m in SETUPS: a = run(o, m, SEED + 1) print(f" {name:<29} {a[0]:.4f} / {a[5]:.4f} / {a[6]:.4f}") ``` ``` setup resp hit prefix hit input total calls stale correct no cache 0.0000 0.0000 44367 45263 240 0.0000 0.9333 prefix cache 0.0000 0.9792 7472 8368 240 0.0000 0.9333 response lifetime 8 0.4708 0.0000 23456 23917 127 0.0208 0.9208 response lifetime 24 0.6750 0.0000 14407 14686 78 0.0500 0.9292 response lifetime 64 0.8000 0.0000 8870 9042 48 0.1333 0.8625 response lifetime unlimited 0.8792 0.0000 5357 5461 29 0.4417 0.5542 prefix + response lifetime 24 0.6750 0.9359 2946 3225 78 0.0500 0.9292 second run (seed 20260219), response hit / stale / correct: no cache 0.0000 / 0.0000 / 0.9083 prefix cache 0.0000 / 0.0000 / 0.9083 response lifetime 8 0.5292 / 0.0292 / 0.8667 response lifetime 24 0.6958 / 0.0833 / 0.8250 response lifetime 64 0.8208 / 0.2167 / 0.6833 response lifetime unlimited 0.9000 / 0.5000 / 0.4042 prefix + response lifetime 24 0.6958 / 0.0833 / 0.8250 ``` The second row is the cheapest gain of this lesson and has no catch. The prefix cache, with a 0.9792 hit rate, cuts input tokens from **44367 to 7472**, that is, **5.94 times**. The stale-response share is **0.0000**, and the correct-response share **does not change**. The reason is in the key: the prefix is keyed **by content**. The moment the pool is updated, the prefix text changes too, the match breaks, and the next call is paid in full. A cache goes stale when its key comes apart from the content. The response-cache rows are the opposite. The key is the **question** text, and the question does not change; what changes is the document the question rests on. With a lifetime of eight steps the hit rate is 0.4708 and the stale share 0.0208; with an unlimited lifetime the hit rate rises to 0.8792 and the stale share climbs to **0.4417**. Calls fall from 240 to 29, and input tokens fall to 5357. From the cost side, the last row is the best row. From the product side, the correct-response share has fallen **from 0.9333 to 0.5542**: in about one of every two requests, the user is told a rule that is no longer valid. ## Choosing the Lifetime Is an Accuracy Decision Where the loss becomes measurable is read from the run-to-run difference. The no-cache baseline gives 0.9083 on the second seed, so the run-to-run difference is **0.0250**. At a lifetime of eight the loss is 0.0125, at twenty-four 0.0041 — both below the run-to-run difference and **counted as unmeasured**. At a lifetime of sixty-four the loss is 0.0708 and at unlimited 0.3791; these are about three and fifteen times the run-to-run difference respectively, that is, **measured**. The decision can be read from here: under this update schedule, extending the lifetime up to twenty-four costs nothing, while raising it to sixty-four does not. This reading carries a warning. In the second run the loss at lifetime twenty-four rises to 0.0833, that is, three times the run-to-run difference relative to its own baseline. The same setting is below the noise in one run and measurable in the other. This comes from twenty-four steps being **close to the update interval**: since the pool changes every twenty-four steps, fitting the lifetime exactly to that interval leaves the decision to the timing luck of the document update. The cache lifetime cannot be chosen **independently** of how often the data changes. The last row is the setup in which the two caches work together, and it produces a number that neither gives alone: input tokens of **2946**, that is, a **15.06-times** reduction from the baseline, with a stale share identical to lifetime twenty-four's, 0.0500. The same row holds a subtler effect: the prefix hit rate falls from 0.9792 to **0.9359**. Because the response cache cuts the number of calls from 240 to 78, the matches broken by the four updates are now divided by a much smaller divisor. One cache improves the other's hit rate by **lowering** it; the numbers of the two scaffoldings cannot be measured separately and added up. ## Summary - Of the prompt's 186 tokens, 175 are shared by every request; which cache will pay off cannot be known until the size of the repetition is measured. - The prefix cache is keyed by content, so it does not go stale: with a 0.9792 hit rate it cuts input from 44367 to 7472 and leaves the correct-response share unchanged. - In the response cache, as the lifetime grows the hit rate rises from 0.4708 to 0.8792, while the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. - The run-to-run difference is 0.0250: the losses at lifetimes eight and twenty-four stay below this difference and count as unmeasured, while the losses at sixty-four and unlimited are measured. - When the lifetime is chosen close to the update interval, the decision becomes unstable; the same setting loses nothing above the noise in one run and three times it in the other. - The two caches together cut input to 2946, but the response cache lowers the prefix hit rate from 0.9792 to 0.9359: the numbers of the scaffoldings cannot be measured separately and added up. ## Next Step What this lesson gained is tokens. A response returned from the cache makes no call, so it also shortens the user's wait, but this lesson did not count that side at all: the table carries only token, call and accuracy columns. Yet in a support assistant the most visible measure is waiting time, and what determines it is not the cache alone. The size of the context, whether the call is made alone or as part of a chain, and whether the expensive step runs on every request or only when needed each change the wait separately. The next lesson compares these three setups: which **latency class** the small context, the large context and the two-stage call sit in, and what each one's accuracy is.