--- title: 'Rate Limits and Retries' source: 'https://academia.sh/en/courses/model-apis/rate-limits-and-retries' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts the rate limit, the timeout and the partial output as separate failure classes and writes the token cost of each.' - 'Compares fixed-interval and backoff retry on the same workload at two load levels.' - 'Shows with numbers how retry multiplies the call count, the tokens and the rate limit rejections.' - 'Measures the harm that resilience scaffolding does to another flow sharing the same limit.' --- # Rate Limits and Retries Running the same 160 fictional tickets under three policies at two load levels: the bare client completes 0.8562 at both loads and never hits the rate limit. At light load retry raises completion to 1.0000, at a cost of 25 extra calls and 1209 tokens. At heavy load the same scaffolding does damage: fixed-interval retry falls to 0.8000 and produces 151 rate limit rejections, while backoff retry stays 0.0188 above the bare client at 0.8750, but the run-to-run difference is also 0.0188. The interactive flow that shares the same limit drops from 0.9500 to 0.8667: retry rescues its own work while breaking someone else's. The previous eight lessons measured what the API offers: carrying conversation state, streaming, tool calls, the structured output mode, multimodal input, the embedding API, the compatible API layer and the batch call. All of them shared one silent assumption: **the call succeeded.** Every token counted was a token that was paid for and received in return. In a production pipeline this assumption does not hold. The call is rejected, cut off midway, or never answers. The remedy for these is a known procedure: retry. Retry was measured in the Observability and Reliability course and in the Resilience and Reliability course; its procedure is not repeated here. The only thing added in this lesson comes from the call being **expensive and uncertain**: every attempt pays for the input once more, every attempt loads one more call onto the same rate limit, and an output cut off midway has spent tokens without being of any use. That is the problem itself — retry feeds the source of what it fixes. - **PQ1.** The code is a **simulator**; no real endpoint is called. The tickets, the templates and the network behavior are **fictional**, the seed is **20260218**. - **PQ2.** Time is counted in **steps**; raw time is not written. One step is one sending round of the client. - **PQ3. The rate limit is read over a sliding window, and rejected calls are counted too:** if the total number of calls made in the last three steps exceeds twenty-four, the call is rejected. On an endpoint that does not count rejections the difference measured below shrinks; this behavior is an assumption and it determines the result. - **PQ4.** A rejected call **spends no tokens**. Every accepted call pays its input in full. - **PQ5.** There are two content failures: the **timeout** (cut off before the output starts, the input is paid) and the **partial output** (the input plus one output token is paid, the response cannot be read). - **PQ6.** The client sends at most twelve calls in a step; a request is tried at most four times. - **PQ7. Two workloads share the same limit:** batch classification of 160 records, and an **interactive** status query that arrives every other step. The interactive query's latency budget is one step. - **PQ8.** Load is the number of requests arriving per step during the burst: **three for light, six for heavy**. ## Three Failures of a Call and Their Cost The simulator carries one more thing in this topic: **network behavior**. The same prompt can be rejected depending on how full the window is; even when it is accepted, the response can be cut off midway. The three classes need to be counted separately, because **their costs differ**. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the network behavior is also # written in the lesson. The tickets and templates are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF QUOTA, WINDOW = 24, 3 # calls counted in the sliding window / window steps CONCURRENCY, HORIZON, MAX_TRIES = 12, 140, 4 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] EXTRAS = ["ayrica gecen ay tarife degisikligi de yapilmisti", "kayit numarasini ekliyorum ekip hala gelmedi", "odeme gorunmuyor ama dekont elimde duruyor", "duyurulari takip ediyorum yine de bilgi alamadim"] TICKETS = [] for i in range(160): # 160 FICTIONAL support tickets r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = [x for x in TEMPLATES if x[1] == k][int(r() * 3)][0] TICKETS.append({"no": 4001 + i, "topic": k, "text": m + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "dekon": "odeme", "sikay": "sikayet"} INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . ") SERVER = {"history": []} def prompt_of(t): return INSTRUCTION + f"Talep {t['no']} : {t['text']}" def label_of(tok): # most likely choice; what narrows the candidates is the label set candidates = [k for k in CONCEPTS if k[:5] in tok] p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if CUES.get(t) in p: p[CUES[t]] += 1.0 return max(sorted(p), key=lambda a: p[a]) def call(prompt, step, attempt=0, seed=SEED): """FICTIONAL endpoint. The rate limit counts in a sliding window, REJECTED CALLS INCLUDED.""" g = SERVER["history"] while len(g) <= step: g.append(0) g[step] += 1 if sum(g[max(0, step - WINDOW + 1):step + 1]) > QUOTA: return {"status": "rate_limit", "input": 0, "output": 0, "reply": ""} tok = tokenize(prompt) r = rng((seed ^ trace_id(prompt)) + 7919 * attempt) if r() < 0.03 + 0.0006 * len(tok): # cut off before the output started return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""} k = label_of(tok) if r() < 0.09: # output cut off midway return {"status": "partial_output", "input": len(tok), "output": 1, "reply": k[:2]} return {"status": "ok", "input": len(tok), "output": len(tokenize(k)), "reply": k} SERVER["history"] = [] print("single call :", call(prompt_of(TICKETS[0]), 0)) SERVER["history"] = [QUOTA] # window already full print("full window :", call(prompt_of(TICKETS[1]), 0)) SERVER["history"] = [] classes = {} for t in TICKETS: SERVER["history"] = [] y = call(prompt_of(t), 0) a, b = classes.get(y["status"], (0, 0)) classes[y["status"]] = (a + 1, b + y["input"] + y["output"]) print(f"\n{'class':<14} {'count':>5} {'tokens':>8} outcome") for d in ("ok", "timeout", "partial_output"): a, b = classes[d] print(f"{d:<14} {a:>5} {b:>8} {'usable' if d == 'ok' else 'discarded'}") ``` ``` single call : {'status': 'ok', 'input': 44, 'output': 2, 'reply': 'kesinti'} full window : {'status': 'rate_limit', 'input': 0, 'output': 0, 'reply': ''} class count tokens outcome ok 137 6523 usable timeout 9 436 discarded partial_output 14 655 discarded ``` In the last table the window was emptied before every call, that is, the rate limit is switched off and only the content failures remain. Twenty-three requests are lost, and **1091 tokens of these requests were paid for and nothing was received in return**. Given that the total is 7614 tokens, the share wasted is **0.1433**. This is the price paid by a single-attempt client, without any retry being made. The two failure classes do not cost the same. The timeout burns 436 tokens over nine requests; the partial output 655 over fourteen. The rejected call, on the other hand, spends zero tokens — in payment terms it is the cheapest failure. This ordering says why a retry policy cannot be set up blindly: sending a rejected call again costs only one call, while sending a cut-off call again makes you pay the whole input a second time. ## Three Policies Under Load Now the same 160 tickets are run along a time axis. During the first hundred steps one request arrives per step; at the thirtieth step a **burst** begins and sixty requests pile up either three at a time or six at a time. The interactive status query coming from the field application every other step also uses the same rate limit. Three policies are compared: no retry, **fixed-interval** (one step after every failure) and **backoff** (after one, two, four steps). ```python def run(policy, load, seed=SEED): """policy: none | fixed | backoff . load: requests per step during the burst.""" SERVER["history"] = [] arrivals = [[] for _ in range(HORIZON)] for i in range(len(TICKETS)): arrivals[i if i < 100 else 30 + (i - 100) // load].append(i) queue, active = [], [] s = dict.fromkeys(("calls", "rate", "content", "tok", "wasted", "done", "correct", "idone", "itotal"), 0) for step in range(HORIZON): if step % 2 == 0 and step < 120: # interactive flow active.append({"step": step, "attempt": 0}) s["itotal"] += 1 for i in arrivals[step]: queue.append({"i": i, "ready": step, "attempt": 0}) for x in list(active): if x["step"] + 1 < step: # latency budget used up active.remove(x) continue y = call(f"Saha uygulamasi sorgusu {x['step']} : acik kayit sayisi", step, x["attempt"], seed) s["tok"] += y["input"] + y["output"] if y["status"] == "ok": s["idone"] += 1 active.remove(x) else: s["wasted"] += y["input"] + y["output"] x["attempt"] += 1 for x in [x for x in queue if x["ready"] <= step][:CONCURRENCY]: queue.remove(x) t = TICKETS[x["i"]] y = call(prompt_of(t), step, x["attempt"], seed) s["calls"] += 1 s["tok"] += y["input"] + y["output"] if y["status"] == "ok": s["done"] += 1 s["correct"] += y["reply"] == t["topic"] continue s["wasted"] += y["input"] + y["output"] s["rate" if y["status"] == "rate_limit" else "content"] += 1 x["attempt"] += 1 if policy == "none" or x["attempt"] >= MAX_TRIES: continue x["ready"] = step + (1 if policy == "fixed" else 2 ** (x["attempt"] - 1)) queue.append(x) n = len(TICKETS) return (s["done"] / n, s["idone"] / s["itotal"], s["calls"], s["rate"], s["content"], s["tok"], s["wasted"], s["correct"] / n) print("load policy completed interactive calls rate limit content error" " tokens wasted correct") for load, name in ((3, "light"), (6, "heavy")): for p in ("none", "fixed", "backoff"): a = run(p, load) print(f"{name} {p:<8} {a[0]:>10.4f} {a[1]:>11.4f} {a[2]:>5} {a[3]:>10}" f" {a[4]:>13} {a[5]:>8} {a[6]:>10} {a[7]:>7.4f}") print("\nsecond run (seed 20260219), completed / interactive:") for load, name in ((3, "light"), (6, "heavy")): b = [run(p, load, SEED + 1) for p in ("none", "fixed", "backoff")] print(f" {name} " + " ".join(f"{p}: {x[0]:.4f} / {x[1]:.4f}" for p, x in zip(("none", "fixed", "backoff"), b))) ``` ``` load policy completed interactive calls rate limit content error tokens wasted correct light none 0.8562 0.9500 160 0 23 8425 1218 0.8438 light fixed 1.0000 0.9500 185 0 25 9634 1311 0.9875 light backoff 1.0000 0.9500 185 0 25 9634 1311 0.9875 heavy none 0.8562 0.9500 160 0 23 8425 1218 0.8438 heavy fixed 0.8000 0.8667 295 151 16 7658 873 0.7937 heavy backoff 0.8750 0.8833 289 129 20 8389 1038 0.8625 second run (seed 20260219), completed / interactive: light none: 0.8750 / 1.0000 fixed: 1.0000 / 1.0000 backoff: 1.0000 / 1.0000 heavy none: 0.8750 / 1.0000 fixed: 0.6813 / 0.8667 backoff: 0.7188 / 0.8833 ``` The first row of the table to be read is that the bare client is **the same at both loads**: 0.8562 completion, **zero rate limit rejections**. The burst rising from three to six per step changes nothing for the bare client, because the incoming load still stays below the limit. The limit will be exceeded not by the workload but **by the scaffolding**. At light load retry works exactly as expected: completion rises from 0.8562 to **1.0000**, and the two policies are **identical to each other** — because when there is no rejection at all, backoff has nothing to delay. The cost is plain: 25 extra calls, 1209 extra tokens, and wasted tokens from 1218 to 1311. That is, for the twenty-three requests rescued, part of the tokens paid went to waste again, because some retries failed too. ## Whose Work Retry Breaks The heavy-load rows are the clearest example of the course's second claim. Fixed-interval retry **lowers completion from 0.8562 to 0.8000**. Scaffolding was added, the number went down. The reason sits in the columns: calls rise from 160 to 295, and **151 of these calls are rejected by the rate limit**. Because rejected calls are also counted in the window, every rejection makes the next rejection easier; the call sent to fix the problem enlarges the problem to be fixed. While content errors fall from 23 to 16, total completion falls too: requests are no longer lost because of their content but **because of the congestion created by their own attempts**. Backoff retry gives 0.8750 under the same load, that is, **0.0188** above the bare client. The second run must be set beside this number: the bare client gives 0.8750 on seed 20260219, and the run-to-run difference is exactly **0.0188**. The gain does not exceed the run-to-run difference, therefore it **counts as unmeasured**. By contrast, the loss of fixed-interval retry is 0.0562, three times the run-to-run difference — the loss is measured. The heavy-load rows of the second run confirm this: there both policies fall below the bare client (0.6813 and 0.7188). The last column is in the place that most measurements never look at. The interactive flow shares the same rate limit and has no retry of its own; its latency budget is one step. With the bare client this flow completes 0.9500. When retry is added to the batch work it drops to **0.8667**. The interactive flow changed nothing, its requests did not increase, its prompt did not grow; only **its neighbor became resilient**. This is the most insidious cost of resilience scaffolding: while rescuing its own measure it consumes a shared resource, and the harm shows up **in another measure**. If a retry policy is evaluated only by its own failure share, this row is never seen. ## Summary - The three failures of a call carry separate costs: a rate limit rejection spends no tokens, a timeout pays the input, a partial output pays the input plus the partial output; the share of tokens wasted in a single-attempt client is 0.1433. - The bare client gives the same result at light and heavy load (0.8562) and never hits the rate limit; the limit is exceeded not by the workload but by the added scaffolding. - At light load retry raises completion to 1.0000 and there is no difference between fixed-interval and backoff; the cost is 25 extra calls and 1209 tokens. - At heavy load fixed-interval retry lowers completion to 0.8000 and produces 151 rate limit rejections: the call sent to fix the problem enlarges the problem to be fixed. - The gain of backoff retry at heavy load is 0.0188, and the run-to-run difference is 0.0188; the gain counts as unmeasured, whereas the 0.0562 loss of fixed-interval retry is measured. - The interactive flow sharing the same limit falls from 0.9500 to 0.8667; the resilience scaffolding leaves the harm in another measure while rescuing its own. ## Next Step In this lesson every call was sent from scratch. Yet all 160 prompts that entered the table begin with the same instruction, and that instruction is tokenized again and paid for again every time. Retry enlarges this repetition by one more layer: at heavy load all 295 calls carry the same shared text. The repetition itself is a cost, and this cost can be lowered by storing the response or the shared context. But a stored response has an age: if the document it rests on has changed, the response is still fast, still cheap and now wrong. The next lesson puts two numbers side by side: how far tokens fall as the cache hit rate rises, and how far the **stale-response share** grows.