--- title: 'Error and Outage Management' source: 'https://academia.sh/en/courses/model-apis/error-and-outage-management' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Separates with numbers how total accuracy shows an improvement or a drop depending on the baseline it is read against.' - 'Shows how writing response provenance to the ledger makes the quality drop visible, by separating the accuracy of the primary tier from that of the fallback tier.' - 'Measures at which point the drop in total accuracy exceeds the run-to-run difference as the outage share grows.' --- # Error and Outage Management Measuring what dropping a request to a fallback tier does to quality when the primary endpoint does not respond: in the setup without fallback the outage turns 26 requests into errors, while in the setup with fallback errors fall to 2 and the total accuracy rises from 0.8450 to 0.8850, that is, the fallback looks like an improvement. Read against the no-outage baseline, the same number has fallen from 0.8975 to 0.8850, and the drop is 0.0125, below the run-to-run difference of 0.0200. When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667. When the outage share is raised from 0.00 to 0.35, the drop in total accuracy exceeds the run-to-run difference only at 0.18; the accuracy of the primary tier does not move across the four rows, what moves is the mix. The ledger of the previous lesson wrote one row for every request: how many calls, how many tokens, correct or not. The only thing not in that ledger was **where the response came from**. When an endpoint does not respond, a second route is opened so that the request is not lost, and this route writes a row in the ledger that looks the same as the first. The circuit breaker, graceful degradation and failover were set up in the Resilience and Reliability course: when to flip the switch, how long an open circuit stays open and the procedure of the half-open trial were measured there. They are not repeated here. The only thing added to this lesson is this: for a language model, the fallback is **not another server doing the same job**. It is a smaller, narrower and less accurate component; and it does this without telling anyone. - **PQ38.** The code is a **simulator**; no real endpoint is called. The tickets, the conversations and the outage window are **fictional**, the seed is **20260218**. - **PQ39.** The workload is the same as in the previous lesson: 400 requests from 154 fictional conversations. - **PQ40. The fallback tier differs in three places:** its context window is **120** instead of 240, **eleven** of the twenty-three entries of the cue table are absent from it, and the label set written into the prompt does **not narrow** its candidate set. All three are known limits of a real small model. - **PQ41.** The outage is set up as a **window**: starting from request 160, for a given share of the workload, the primary tier rejects 0.70 of the requests. - **PQ42.** Outside the outage too, the primary tier fails to answer a share of requests: rate limit, timeout and context overflow. These bring the fallback into play **even without an outage**. - **PQ43. Three setups are compared:** one that turns the dropped request into an error, one that routes it to a human line, and one that drops it to the fallback. All three work on the output of **the same primary run**, so the set of dropped requests is exactly the same in all three. - **PQ44.** Accuracy is counted separately in three places: **total** over all requests, **primary** over what the primary tier answered, **fallback** over what the fallback answered. ## The Silent Cost of the Fallback ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations # and the outage window are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per token b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme", "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"} FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi", "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek", "kaydiniz alindi ilgili birim sizinle iletisime gececek"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the call count return {"status": "rate_limit", "input": len(tok), "output": 0, "reply": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""} candidates = [k for k in CONCEPTS if k[:5] in tok] # the label set narrows the candidates p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)} for t in tok: if CUES.get(t) in p: p[CUES[t]] += 1.0 if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: w = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k) out = tokenize(y) if len(out) > max_output: # stopping condition out, y = out[:max_output], " ".join(out[:max_output]) return {"status": "ok", "input": len(tok), "output": len(out), "reply": y} def read_label(y): return next((s for s in y.split() if s in TOPICS), None) INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: ") DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir", "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir", "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir", "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir", "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"] CONVERSATIONS, i = [], 0 while sum(len(c["turns"]) for c in CONVERSATIONS) < 400: # FICTIONAL workload: conversations r = rng(SEED + 900 + i) i += 1 k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])] CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]), "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]} for t in range(u)]}) REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]} for c in CONVERSATIONS for t in range(len(c["turns"]))][:400] def context(x): return "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . " for o in x["conv"]["turns"][:x["pos"]]) \ + "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . " for j in range(x["conv"]["doc"])) FALLBACK_CUES = {a: b for a, b in CUES.items() if a not in ("tutar", "ayrin", "bozuk", "akmiy", "endek", "okuma", "basam", "abone", "duyur", "mahal", "calis")} FALLBACK_WINDOW = 120 def ask_fallback(prompt, seed=SEED): """Fallback tier: narrow window, few cues, does not narrow the label set.""" COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > FALLBACK_WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""} p = {a: 0.15 for a in CONCEPTS} for t in tok: if FALLBACK_CUES.get(t) in p: p[FALLBACK_CUES[t]] += 1.0 w = [math.exp(p[a] / 1.3) for a in sorted(p)] r = rng((seed + 5) ^ trace_id(prompt)) return {"status": "ok", "input": len(tok), "output": 1, "reply": sorted(p)[pick(r(), [x / sum(w) for x in w])]} def primary(share=0.06, seed=SEED): """Outage window: starting from request 160, the given share of the workload.""" COUNTER[0] = 0 cut, result = int(400 * share), [] for i, x in enumerate(REQUESTS): prompt = INSTRUCTION + context(x) + x["ticket"]["text"] if 160 <= i < 160 + cut and rng(seed ^ trace_id(prompt))() < 0.70: s = {"status": "outage", "input": len(tokenize(prompt)), "output": 0, "reply": ""} COUNTER[0] += 1 else: s = call(prompt, seed) result.append((x, prompt, s)) return result def apply_setup(mode, raw): """The ledger also writes response provenance: primary and fallback are counted apart.""" n = dict(dropped=0, to_fallback=0, error=0, human=0, p_ok=0, p_n=0, f_ok=0, f_n=0, tok=0) for x, prompt, s in raw: n["tok"] += s["input"] + s["output"] if s["status"] == "ok": n["p_n"] += 1 n["p_ok"] += read_label(s["reply"]) == x["ticket"]["topic"] continue n["dropped"] += 1 if mode == "no_fallback": n["error"] += 1 elif mode == "to_human": n["human"] += 1 else: y = ask_fallback(prompt) n["tok"] += y["input"] + y["output"] if y["status"] != "ok": n["error"] += 1 else: n["to_fallback"] += 1 n["f_n"] += 1 n["f_ok"] += read_label(y["reply"]) == x["ticket"]["topic"] n["total"] = (n["p_ok"] + n["f_ok"]) / 400 n["primary_acc"] = n["p_ok"] / max(n["p_n"], 1) n["fallback_acc"] = n["f_ok"] / max(n["f_n"], 1) return n print(f"{'setup':<22} {'dropped':>7} {'error':>5} {'to fallback':>11} {'to human':>8}" f" {'total':>7} {'primary':>7} {'fallback':>8} {'tokens':>7}") for name, mode, raw in (("no outage, no fallback", "no_fallback", primary(0.0)), ("outage, no fallback", "no_fallback", primary()), ("outage, to human", "to_human", primary()), ("outage, fallback", "fallback", primary()), ("no outage, fallback", "fallback", primary(0.0))): a = apply_setup(mode, raw) print(f"{name:<22} {a['dropped']:>7} {a['error']:>5} {a['to_fallback']:>11}" f" {a['human']:>8} {a['total']:>7.4f} {a['primary_acc']:>7.4f}" f" {a['fallback_acc']:>8.4f} {a['tok']:>7}") ``` ``` setup dropped error to fallback to human total primary fallback tokens no outage, no fallback 13 13 0 0 0.8750 0.9044 0.0000 31569 outage, no fallback 26 26 0 0 0.8450 0.9037 0.0000 31548 outage, to human 26 0 0 26 0.8450 0.9037 0.0000 31548 outage, fallback 26 2 24 0 0.8850 0.9037 0.6667 33775 no outage, fallback 13 1 12 0 0.8975 0.9044 0.7500 32726 ``` Read the second and fourth rows side by side, and the fallback looks like a gain. During the outage the setup without fallback turns twenty-six requests into errors and leaves the total accuracy at **0.8450**; in the setup with fallback errors fall to **2** and total accuracy rises to **0.8850**. The difference is **0.0400**, at a cost of **2227 tokens**. On a dashboard this row is read as proof that the scaffolding works. The fifth row sets the same number against another baseline. Had there been no outage at all, total accuracy would have been **0.8975**. The outage happened and the number fell to **0.8850**: a drop of **0.0125**. As will be seen in the next section, the run-to-run difference of this measure is **0.0200**. That is, the drop stayed below the measurement noise. The outage happened, twenty-four requests were answered by another model, and total accuracy did not show it. Only the last two columns say where the drop is. The accuracy of the primary tier is **0.9037**, that of the requests that fell to the fallback **0.6667**. The **0.2371** between them is not a small difference; in one of every three requests the label is wrong. All that was needed for this number to appear was to add one field to the ledger: response provenance, that is, **which tier** it came from. That field holds zero tokens. The third row measures another option. When the dropped request is routed to the human line instead of being given to the fallback, total accuracy stays at **0.8450**, the same as in the setup without fallback; but the error returned to the user is **zero**, and the cost is written to **a countable place** as twenty-six requests' worth of human work. The fallback hides, the human line counts. The choice between these two forms of graceful degradation is about whether making the hidden cost visible is worth it. Finally, the narrow window of the fallback leaves two requests unanswerable. These are the longest conversations: since the window of the fallback tier is 120 tokens, it cannot carry the context the primary tier can carry. The requests that fall to the fallback are not random; **those with the longest context struggle most**. ## As the Outage Share Grows That a drop is not visible does not mean the drop is small. The block below raises the outage share from zero to 0.35 and at every step measures total accuracy with two seeds. ```python baseline = apply_setup("fallback", primary(0.0))["total"] print(f"{'outage':<8} {'fallback share':>14} {'total A':>9} {'total B':>9} {'run-to-run':>10}" f" {'drop':>6} {'visible':<7} {'primary':>7} {'fallback':>8} {'sample':>6}") for share in (0.0, 0.06, 0.18, 0.35): a = apply_setup("fallback", primary(share)) b = apply_setup("fallback", primary(share, SEED + 1)) f, d = abs(a["total"] - b["total"]), baseline - a["total"] print(f"{share:<8.2f} {a['to_fallback'] / 400:>14.4f} {a['total']:>9.4f} {b['total']:>9.4f}" f" {f:>10.4f} {d:>6.4f} {'yes' if d > f else 'no':<7}" f" {a['primary_acc']:>7.4f} {a['fallback_acc']:>8.4f} {a['f_n']:>6}") ``` ``` outage fallback share total A total B run-to-run drop visible primary fallback sample 0.00 0.0300 0.8975 0.8725 0.0250 0.0000 no 0.9044 0.7500 12 0.06 0.0600 0.8850 0.8650 0.0200 0.0125 no 0.9037 0.6667 24 0.18 0.1275 0.8500 0.8325 0.0175 0.0475 yes 0.9032 0.6275 51 0.35 0.2350 0.7850 0.7750 0.0100 0.1125 yes 0.8927 0.5957 94 ``` The visibility column flips between the second and third rows. At outage share 0.06 the drop is 0.0125 and the run-to-run difference 0.0200: an alert threshold that looks at total accuracy **sees nothing**. When the outage share rises to 0.18, the drop is 0.0475 and exceeds the run-to-run difference. That is, total accuracy is not a quality indicator but an **indicator that depends on the outage share**; it reports quality only when the outage becomes large enough. The first row is the extreme case of this. Even when there is no outage at all, twelve requests are answered by the fallback, because the rate limit and the timeout produce a share in every run. The accuracy of these twelve requests is **0.7500**; in normal operation, without any event record being written, **0.0300** of the requests are answered by a worse model. The primary accuracy column stays in place in all four rows: **0.9044 / 0.9037 / 0.9032 / 0.8927**. The primary model did not degrade; what degraded is **the mix**. The accuracy of the fallback tier, on the other hand, falls with the share (from 0.7500 to 0.5957), because as the outage grows, the proportion of long-context requests among those that fall to the fallback rises. The last column also writes the limit of this reading: the 0.6667 in the 0.06 row was computed from only twenty-four requests. It is a number with a small sample and cannot be tied to a threshold on its own; but it is **far more informative** than total accuracy, because it measures what it wants to measure. ## Summary - The fallback tier is not another server doing the same job: its window is narrow, its cue table is incomplete, and the label set does not narrow its candidate set. - During the outage the fallback brings errors down from 26 to 2, raises total accuracy from 0.8450 to 0.8850 and adds 2227 tokens; against the no-outage baseline, it brings accuracy down from 0.8975 to 0.8850. - The drop is 0.0125 and the run-to-run difference is 0.0200: an indicator that looks at total accuracy never sees the outage. - When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667, difference 0.2371; the cost of this field is zero tokens. - Total accuracy is not a quality indicator but an indicator that depends on the outage share: the drop exceeds the run-to-run difference only when the outage share reaches 0.18. - The accuracy of the primary tier does not move across the four shares (between 0.9044 and 0.8927); what degrades is the mix, and even with no outage 0.0300 of the requests are answered by the fallback. ## Next Step In this lesson the run-to-run difference was used as a threshold: every difference below 0.0200 was counted as unmeasured. This threshold itself was never questioned. Why do the same workload, the same prompt and the same code give different numbers in two runs, which item moves when it is released, and does the movement stop if everything is pinned? The next lesson counts the items of a model call one by one, releases each of them alone and measures how many responses change, and separates how many of these items are **in our hands**.