Rate Limits and Retries

The previous eight lessons measured what the API offers: carrying conversation state, streaming, tool calls, the structured output mode, multimodal input, the embedding API, the compatible API layer and the batch call. All of them shared one silent assumption: the call succeeded. Every token counted was a token that was paid for and received in return.

In a production pipeline this assumption does not hold. The call is rejected, cut off midway, or never answers. The remedy for these is a known procedure: retry. Retry was measured in the Observability and Reliability course and in the Resilience and Reliability course; its procedure is not repeated here. The only thing added in this lesson comes from the call being expensive and uncertain: every attempt pays for the input once more, every attempt loads one more call onto the same rate limit, and an output cut off midway has spent tokens without being of any use. That is the problem itself — retry feeds the source of what it fixes.

• PQ1. The code is a simulator; no real endpoint is called. The tickets, the templates and the network behavior are fictional, the seed is 20260218.
• PQ2. Time is counted in steps; raw time is not written. One step is one sending round of the client.
• PQ3. The rate limit is read over a sliding window, and rejected calls are counted too: if the total number of calls made in the last three steps exceeds twenty-four, the call is rejected. On an endpoint that does not count rejections the difference measured below shrinks; this behavior is an assumption and it determines the result.
• PQ4. A rejected call spends no tokens. Every accepted call pays its input in full.
• PQ5. There are two content failures: the timeout (cut off before the output starts, the input is paid) and the partial output (the input plus one output token is paid, the response cannot be read).
• PQ6. The client sends at most twelve calls in a step; a request is tried at most four times.
• PQ7. Two workloads share the same limit: batch classification of 160 records, and an interactive status query that arrives every other step. The interactive query’s latency budget is one step.
• PQ8. Load is the number of requests arriving per step during the burst: three for light, six for heavy.

Three Failures of a Call and Their Cost

The simulator carries one more thing in this topic: network behavior. The same prompt can be rejected depending on how full the window is; even when it is accepted, the response can be cut off midway. The three classes need to be counted separately, because their costs differ.

# SIMULATOR -- FICTIONAL. No real endpoint is called; the network behavior is also
# written in the lesson. The tickets and templates are fictional, seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
QUOTA, WINDOW = 24, 3        # calls counted in the sliding window / window steps
CONCURRENCY, HORIZON, MAX_TRIES = 12, 140, 4


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):             # FICTIONAL tokenizer, at most five characters per token
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
TEMPLATES = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
             ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
             ("fatura ayrintisini nereden gorebilirim", "fatura"),
             ("evde su akmiyor bir ariza var galiba", "ariza"),
             ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
             ("ariza kaydi actim ama ekip gelmedi", "ariza"),
             ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
             ("sayac degisimi icin randevu istiyorum", "sayac"),
             ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
             ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
             ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
             ("birim ucret abone tipine gore degisiyor mu", "tarife"),
             ("mahallede kesinti var ne zaman gelecek", "kesinti"),
             ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
             ("aksamdan beri kesinti yasiyoruz", "kesinti")]
EXTRAS = ["ayrica gecen ay tarife degisikligi de yapilmisti",
          "kayit numarasini ekliyorum ekip hala gelmedi",
          "odeme gorunmuyor ama dekont elimde duruyor",
          "duyurulari takip ediyorum yine de bilgi alamadim"]
TICKETS = []
for i in range(160):         # 160 FICTIONAL support tickets
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = [x for x in TEMPLATES if x[1] == k][int(r() * 3)][0]
    TICKETS.append({"no": 4001 + i, "topic": k,
                    "text": m + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife",
        "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti",
        "duyur": "kesinti", "mahal": "kesinti", "dekon": "odeme", "sikay": "sikayet"}
INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , "
               "kesinti . Yanit olarak tek etiket yaz . ")
SERVER = {"history": []}


def prompt_of(t):
    return INSTRUCTION + f"Talep {t['no']} : {t['text']}"


def label_of(tok):           # most likely choice; what narrows the candidates is the label set
    candidates = [k for k in CONCEPTS if k[:5] in tok]
    p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)}
    for t in tok:
        if CUES.get(t) in p:
            p[CUES[t]] += 1.0
    return max(sorted(p), key=lambda a: p[a])


def call(prompt, step, attempt=0, seed=SEED):
    """FICTIONAL endpoint. The rate limit counts in a sliding window, REJECTED CALLS INCLUDED."""
    g = SERVER["history"]
    while len(g) <= step:
        g.append(0)
    g[step] += 1
    if sum(g[max(0, step - WINDOW + 1):step + 1]) > QUOTA:
        return {"status": "rate_limit", "input": 0, "output": 0, "reply": ""}
    tok = tokenize(prompt)
    r = rng((seed ^ trace_id(prompt)) + 7919 * attempt)
    if r() < 0.03 + 0.0006 * len(tok):                  # cut off before the output started
        return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""}
    k = label_of(tok)
    if r() < 0.09:                                      # output cut off midway
        return {"status": "partial_output", "input": len(tok), "output": 1,
                "reply": k[:2]}
    return {"status": "ok", "input": len(tok), "output": len(tokenize(k)),
            "reply": k}


SERVER["history"] = []
print("single call :", call(prompt_of(TICKETS[0]), 0))
SERVER["history"] = [QUOTA]                             # window already full
print("full window :", call(prompt_of(TICKETS[1]), 0))
SERVER["history"] = []
classes = {}
for t in TICKETS:
    SERVER["history"] = []
    y = call(prompt_of(t), 0)
    a, b = classes.get(y["status"], (0, 0))
    classes[y["status"]] = (a + 1, b + y["input"] + y["output"])
print(f"\n{'class':<14} {'count':>5}  {'tokens':>8}  outcome")
for d in ("ok", "timeout", "partial_output"):
    a, b = classes[d]
    print(f"{d:<14} {a:>5}  {b:>8}  {'usable' if d == 'ok' else 'discarded'}")

single call : {'status': 'ok', 'input': 44, 'output': 2, 'reply': 'kesinti'}
full window : {'status': 'rate_limit', 'input': 0, 'output': 0, 'reply': ''}

class          count    tokens  outcome
ok               137      6523  usable
timeout            9       436  discarded
partial_output    14       655  discarded

In the last table the window was emptied before every call, that is, the rate limit is switched off and only the content failures remain. Twenty-three requests are lost, and 1091 tokens of these requests were paid for and nothing was received in return. Given that the total is 7614 tokens, the share wasted is 0.1433. This is the price paid by a single-attempt client, without any retry being made.

The two failure classes do not cost the same. The timeout burns 436 tokens over nine requests; the partial output 655 over fourteen. The rejected call, on the other hand, spends zero tokens — in payment terms it is the cheapest failure. This ordering says why a retry policy cannot be set up blindly: sending a rejected call again costs only one call, while sending a cut-off call again makes you pay the whole input a second time.

Three Policies Under Load

Now the same 160 tickets are run along a time axis. During the first hundred steps one request arrives per step; at the thirtieth step a burst begins and sixty requests pile up either three at a time or six at a time. The interactive status query coming from the field application every other step also uses the same rate limit. Three policies are compared: no retry, fixed-interval (one step after every failure) and backoff (after one, two, four steps).

def run(policy, load, seed=SEED):
    """policy: none | fixed | backoff . load: requests per step during the burst."""
    SERVER["history"] = []
    arrivals = [[] for _ in range(HORIZON)]
    for i in range(len(TICKETS)):
        arrivals[i if i < 100 else 30 + (i - 100) // load].append(i)
    queue, active = [], []
    s = dict.fromkeys(("calls", "rate", "content", "tok", "wasted", "done", "correct",
                       "idone", "itotal"), 0)
    for step in range(HORIZON):
        if step % 2 == 0 and step < 120:                # interactive flow
            active.append({"step": step, "attempt": 0})
            s["itotal"] += 1
        for i in arrivals[step]:
            queue.append({"i": i, "ready": step, "attempt": 0})
        for x in list(active):
            if x["step"] + 1 < step:                    # latency budget used up
                active.remove(x)
                continue
            y = call(f"Saha uygulamasi sorgusu {x['step']} : acik kayit sayisi",
                     step, x["attempt"], seed)
            s["tok"] += y["input"] + y["output"]
            if y["status"] == "ok":
                s["idone"] += 1
                active.remove(x)
            else:
                s["wasted"] += y["input"] + y["output"]
                x["attempt"] += 1
        for x in [x for x in queue if x["ready"] <= step][:CONCURRENCY]:
            queue.remove(x)
            t = TICKETS[x["i"]]
            y = call(prompt_of(t), step, x["attempt"], seed)
            s["calls"] += 1
            s["tok"] += y["input"] + y["output"]
            if y["status"] == "ok":
                s["done"] += 1
                s["correct"] += y["reply"] == t["topic"]
                continue
            s["wasted"] += y["input"] + y["output"]
            s["rate" if y["status"] == "rate_limit" else "content"] += 1
            x["attempt"] += 1
            if policy == "none" or x["attempt"] >= MAX_TRIES:
                continue
            x["ready"] = step + (1 if policy == "fixed" else 2 ** (x["attempt"] - 1))
            queue.append(x)
    n = len(TICKETS)
    return (s["done"] / n, s["idone"] / s["itotal"], s["calls"], s["rate"],
            s["content"], s["tok"], s["wasted"], s["correct"] / n)


print("load   policy    completed  interactive  calls  rate limit  content error"
      "    tokens      wasted  correct")
for load, name in ((3, "light"), (6, "heavy")):
    for p in ("none", "fixed", "backoff"):
        a = run(p, load)
        print(f"{name}  {p:<8} {a[0]:>10.4f}  {a[1]:>11.4f}  {a[2]:>5}  {a[3]:>10}"
              f"  {a[4]:>13}  {a[5]:>8}  {a[6]:>10}  {a[7]:>7.4f}")
print("\nsecond run (seed 20260219), completed / interactive:")
for load, name in ((3, "light"), (6, "heavy")):
    b = [run(p, load, SEED + 1) for p in ("none", "fixed", "backoff")]
    print(f"  {name} " + "  ".join(f"{p}: {x[0]:.4f} / {x[1]:.4f}"
                                   for p, x in zip(("none", "fixed", "backoff"), b)))

load   policy    completed  interactive  calls  rate limit  content error    tokens      wasted  correct
light  none         0.8562       0.9500    160           0             23      8425        1218   0.8438
light  fixed        1.0000       0.9500    185           0             25      9634        1311   0.9875
light  backoff      1.0000       0.9500    185           0             25      9634        1311   0.9875
heavy  none         0.8562       0.9500    160           0             23      8425        1218   0.8438
heavy  fixed        0.8000       0.8667    295         151             16      7658         873   0.7937
heavy  backoff      0.8750       0.8833    289         129             20      8389        1038   0.8625

second run (seed 20260219), completed / interactive:
  light none: 0.8750 / 1.0000  fixed: 1.0000 / 1.0000  backoff: 1.0000 / 1.0000
  heavy none: 0.8750 / 1.0000  fixed: 0.6813 / 0.8667  backoff: 0.7188 / 0.8833

The first row of the table to be read is that the bare client is the same at both loads: 0.8562 completion, zero rate limit rejections. The burst rising from three to six per step changes nothing for the bare client, because the incoming load still stays below the limit. The limit will be exceeded not by the workload but by the scaffolding.

At light load retry works exactly as expected: completion rises from 0.8562 to 1.0000, and the two policies are identical to each other — because when there is no rejection at all, backoff has nothing to delay. The cost is plain: 25 extra calls, 1209 extra tokens, and wasted tokens from 1218 to 1311. That is, for the twenty-three requests rescued, part of the tokens paid went to waste again, because some retries failed too.

Whose Work Retry Breaks

The heavy-load rows are the clearest example of the course’s second claim. Fixed-interval retry lowers completion from 0.8562 to 0.8000. Scaffolding was added, the number went down. The reason sits in the columns: calls rise from 160 to 295, and 151 of these calls are rejected by the rate limit. Because rejected calls are also counted in the window, every rejection makes the next rejection easier; the call sent to fix the problem enlarges the problem to be fixed. While content errors fall from 23 to 16, total completion falls too: requests are no longer lost because of their content but because of the congestion created by their own attempts.

Backoff retry gives 0.8750 under the same load, that is, 0.0188 above the bare client. The second run must be set beside this number: the bare client gives 0.8750 on seed 20260219, and the run-to-run difference is exactly 0.0188. The gain does not exceed the run-to-run difference, therefore it counts as unmeasured. By contrast, the loss of fixed-interval retry is 0.0562, three times the run-to-run difference — the loss is measured. The heavy-load rows of the second run confirm this: there both policies fall below the bare client (0.6813 and 0.7188).

The last column is in the place that most measurements never look at. The interactive flow shares the same rate limit and has no retry of its own; its latency budget is one step. With the bare client this flow completes 0.9500. When retry is added to the batch work it drops to 0.8667. The interactive flow changed nothing, its requests did not increase, its prompt did not grow; only its neighbor became resilient. This is the most insidious cost of resilience scaffolding: while rescuing its own measure it consumes a shared resource, and the harm shows up in another measure. If a retry policy is evaluated only by its own failure share, this row is never seen.

Summary

• The three failures of a call carry separate costs: a rate limit rejection spends no tokens, a timeout pays the input, a partial output pays the input plus the partial output; the share of tokens wasted in a single-attempt client is 0.1433.
• The bare client gives the same result at light and heavy load (0.8562) and never hits the rate limit; the limit is exceeded not by the workload but by the added scaffolding.
• At light load retry raises completion to 1.0000 and there is no difference between fixed-interval and backoff; the cost is 25 extra calls and 1209 tokens.
• At heavy load fixed-interval retry lowers completion to 0.8000 and produces 151 rate limit rejections: the call sent to fix the problem enlarges the problem to be fixed.
• The gain of backoff retry at heavy load is 0.0188, and the run-to-run difference is 0.0188; the gain counts as unmeasured, whereas the 0.0562 loss of fixed-interval retry is measured.
• The interactive flow sharing the same limit falls from 0.9500 to 0.8667; the resilience scaffolding leaves the harm in another measure while rescuing its own.

Next Step

In this lesson every call was sent from scratch. Yet all 160 prompts that entered the table begin with the same instruction, and that instruction is tokenized again and paid for again every time. Retry enlarges this repetition by one more layer: at heavy load all 295 calls carry the same shared text. The repetition itself is a cost, and this cost can be lowered by storing the response or the shared context. But a stored response has an age: if the document it rests on has changed, the response is still fast, still cheap and now wrong. The next lesson puts two numbers side by side: how far tokens fall as the cache hit rate rises, and how far the stale-response share grows.