Error and Outage Management

The ledger of the previous lesson wrote one row for every request: how many calls, how many tokens, correct or not. The only thing not in that ledger was where the response came from. When an endpoint does not respond, a second route is opened so that the request is not lost, and this route writes a row in the ledger that looks the same as the first.

The circuit breaker, graceful degradation and failover were set up in the Resilience and Reliability course: when to flip the switch, how long an open circuit stays open and the procedure of the half-open trial were measured there. They are not repeated here. The only thing added to this lesson is this: for a language model, the fallback is not another server doing the same job. It is a smaller, narrower and less accurate component; and it does this without telling anyone.

• PQ38. The code is a simulator; no real endpoint is called. The tickets, the conversations and the outage window are fictional, the seed is 20260218.
• PQ39. The workload is the same as in the previous lesson: 400 requests from 154 fictional conversations.
• PQ40. The fallback tier differs in three places: its context window is 120 instead of 240, eleven of the twenty-three entries of the cue table are absent from it, and the label set written into the prompt does not narrow its candidate set. All three are known limits of a real small model.
• PQ41. The outage is set up as a window: starting from request 160, for a given share of the workload, the primary tier rejects 0.70 of the requests.
• PQ42. Outside the outage too, the primary tier fails to answer a share of requests: rate limit, timeout and context overflow. These bring the fallback into play even without an outage.
• PQ43. Three setups are compared: one that turns the dropped request into an error, one that routes it to a human line, and one that drops it to the fallback. All three work on the output of the same primary run, so the set of dropped requests is exactly the same in all three.
• PQ44. Accuracy is counted separately in three places: total over all requests, primary over what the primary tier answered, fallback over what the fallback answered.

The Silent Cost of the Fallback

# SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations
# and the outage window are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per token
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                        "faturami odedim ama sistemde borc gorunuyor",
                        "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                         "planli kesinti duyurusu nereden takip edilir",
                         "aksamdan beri kesinti yasiyoruz"]}
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme",
        "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza",
        "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"}
FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi",
             "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek",
             "kaydiniz alindi ilgili birim sizinle iletisime gececek"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""}
    if COUNTER[0] % 37 == 0:                        # the rate limit depends on the call count
        return {"status": "rate_limit", "input": len(tok), "output": 0, "reply": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""}
    candidates = [k for k in CONCEPTS if k[:5] in tok]      # the label set narrows the candidates
    p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)}
    for t in tok:
        if CUES.get(t) in p:
            p[CUES[t]] += 1.0
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        w = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:                       # stopping condition
        out, y = out[:max_output], " ".join(out[:max_output])
    return {"status": "ok", "input": len(tok), "output": len(out), "reply": y}


def read_label(y):
    return next((s for s in y.split() if s in TOPICS), None)


INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , "
               "kesinti . Yanit olarak tek etiket yaz . Talep: ")
DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir",
             "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir",
             "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir",
             "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir",
             "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"]
CONVERSATIONS, i = [], 0
while sum(len(c["turns"]) for c in CONVERSATIONS) < 400:    # FICTIONAL workload: conversations
    r = rng(SEED + 900 + i)
    i += 1
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])]
    CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]),
                          "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]}
                                    for t in range(u)]})
REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]}
            for c in CONVERSATIONS for t in range(len(c["turns"]))][:400]


def context(x):
    return "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . "
                   for o in x["conv"]["turns"][:x["pos"]]) \
        + "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . "
                  for j in range(x["conv"]["doc"]))


FALLBACK_CUES = {a: b for a, b in CUES.items() if a not in
                 ("tutar", "ayrin", "bozuk", "akmiy", "endek", "okuma", "basam",
                  "abone", "duyur", "mahal", "calis")}
FALLBACK_WINDOW = 120


def ask_fallback(prompt, seed=SEED):
    """Fallback tier: narrow window, few cues, does not narrow the label set."""
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > FALLBACK_WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""}
    p = {a: 0.15 for a in CONCEPTS}
    for t in tok:
        if FALLBACK_CUES.get(t) in p:
            p[FALLBACK_CUES[t]] += 1.0
    w = [math.exp(p[a] / 1.3) for a in sorted(p)]
    r = rng((seed + 5) ^ trace_id(prompt))
    return {"status": "ok", "input": len(tok), "output": 1,
            "reply": sorted(p)[pick(r(), [x / sum(w) for x in w])]}


def primary(share=0.06, seed=SEED):
    """Outage window: starting from request 160, the given share of the workload."""
    COUNTER[0] = 0
    cut, result = int(400 * share), []
    for i, x in enumerate(REQUESTS):
        prompt = INSTRUCTION + context(x) + x["ticket"]["text"]
        if 160 <= i < 160 + cut and rng(seed ^ trace_id(prompt))() < 0.70:
            s = {"status": "outage", "input": len(tokenize(prompt)), "output": 0, "reply": ""}
            COUNTER[0] += 1
        else:
            s = call(prompt, seed)
        result.append((x, prompt, s))
    return result


def apply_setup(mode, raw):
    """The ledger also writes response provenance: primary and fallback are counted apart."""
    n = dict(dropped=0, to_fallback=0, error=0, human=0, p_ok=0, p_n=0, f_ok=0, f_n=0, tok=0)
    for x, prompt, s in raw:
        n["tok"] += s["input"] + s["output"]
        if s["status"] == "ok":
            n["p_n"] += 1
            n["p_ok"] += read_label(s["reply"]) == x["ticket"]["topic"]
            continue
        n["dropped"] += 1
        if mode == "no_fallback":
            n["error"] += 1
        elif mode == "to_human":
            n["human"] += 1
        else:
            y = ask_fallback(prompt)
            n["tok"] += y["input"] + y["output"]
            if y["status"] != "ok":
                n["error"] += 1
            else:
                n["to_fallback"] += 1
                n["f_n"] += 1
                n["f_ok"] += read_label(y["reply"]) == x["ticket"]["topic"]
    n["total"] = (n["p_ok"] + n["f_ok"]) / 400
    n["primary_acc"] = n["p_ok"] / max(n["p_n"], 1)
    n["fallback_acc"] = n["f_ok"] / max(n["f_n"], 1)
    return n


print(f"{'setup':<22} {'dropped':>7} {'error':>5} {'to fallback':>11} {'to human':>8}"
      f" {'total':>7} {'primary':>7} {'fallback':>8} {'tokens':>7}")
for name, mode, raw in (("no outage, no fallback", "no_fallback", primary(0.0)),
                        ("outage, no fallback", "no_fallback", primary()),
                        ("outage, to human", "to_human", primary()),
                        ("outage, fallback", "fallback", primary()),
                        ("no outage, fallback", "fallback", primary(0.0))):
    a = apply_setup(mode, raw)
    print(f"{name:<22} {a['dropped']:>7} {a['error']:>5} {a['to_fallback']:>11}"
          f" {a['human']:>8} {a['total']:>7.4f} {a['primary_acc']:>7.4f}"
          f" {a['fallback_acc']:>8.4f} {a['tok']:>7}")

setup                  dropped error to fallback to human   total primary fallback  tokens
no outage, no fallback      13    13           0        0  0.8750  0.9044   0.0000   31569
outage, no fallback         26    26           0        0  0.8450  0.9037   0.0000   31548
outage, to human            26     0           0       26  0.8450  0.9037   0.0000   31548
outage, fallback            26     2          24        0  0.8850  0.9037   0.6667   33775
no outage, fallback         13     1          12        0  0.8975  0.9044   0.7500   32726

Read the second and fourth rows side by side, and the fallback looks like a gain. During the outage the setup without fallback turns twenty-six requests into errors and leaves the total accuracy at 0.8450; in the setup with fallback errors fall to 2 and total accuracy rises to 0.8850. The difference is 0.0400, at a cost of 2227 tokens. On a dashboard this row is read as proof that the scaffolding works.

The fifth row sets the same number against another baseline. Had there been no outage at all, total accuracy would have been 0.8975. The outage happened and the number fell to 0.8850: a drop of 0.0125. As will be seen in the next section, the run-to-run difference of this measure is 0.0200. That is, the drop stayed below the measurement noise. The outage happened, twenty-four requests were answered by another model, and total accuracy did not show it.

Only the last two columns say where the drop is. The accuracy of the primary tier is 0.9037, that of the requests that fell to the fallback 0.6667. The 0.2371 between them is not a small difference; in one of every three requests the label is wrong. All that was needed for this number to appear was to add one field to the ledger: response provenance, that is, which tier it came from. That field holds zero tokens.

The third row measures another option. When the dropped request is routed to the human line instead of being given to the fallback, total accuracy stays at 0.8450, the same as in the setup without fallback; but the error returned to the user is zero, and the cost is written to a countable place as twenty-six requests’ worth of human work. The fallback hides, the human line counts. The choice between these two forms of graceful degradation is about whether making the hidden cost visible is worth it.

Finally, the narrow window of the fallback leaves two requests unanswerable. These are the longest conversations: since the window of the fallback tier is 120 tokens, it cannot carry the context the primary tier can carry. The requests that fall to the fallback are not random; those with the longest context struggle most.

As the Outage Share Grows

That a drop is not visible does not mean the drop is small. The block below raises the outage share from zero to 0.35 and at every step measures total accuracy with two seeds.

baseline = apply_setup("fallback", primary(0.0))["total"]
print(f"{'outage':<8} {'fallback share':>14} {'total A':>9} {'total B':>9} {'run-to-run':>10}"
      f" {'drop':>6}  {'visible':<7} {'primary':>7} {'fallback':>8} {'sample':>6}")
for share in (0.0, 0.06, 0.18, 0.35):
    a = apply_setup("fallback", primary(share))
    b = apply_setup("fallback", primary(share, SEED + 1))
    f, d = abs(a["total"] - b["total"]), baseline - a["total"]
    print(f"{share:<8.2f} {a['to_fallback'] / 400:>14.4f} {a['total']:>9.4f} {b['total']:>9.4f}"
          f" {f:>10.4f} {d:>6.4f}  {'yes' if d > f else 'no':<7}"
          f" {a['primary_acc']:>7.4f} {a['fallback_acc']:>8.4f} {a['f_n']:>6}")

outage   fallback share   total A   total B run-to-run   drop  visible primary fallback sample
0.00             0.0300    0.8975    0.8725     0.0250 0.0000  no       0.9044   0.7500     12
0.06             0.0600    0.8850    0.8650     0.0200 0.0125  no       0.9037   0.6667     24
0.18             0.1275    0.8500    0.8325     0.0175 0.0475  yes      0.9032   0.6275     51
0.35             0.2350    0.7850    0.7750     0.0100 0.1125  yes      0.8927   0.5957     94

The visibility column flips between the second and third rows. At outage share 0.06 the drop is 0.0125 and the run-to-run difference 0.0200: an alert threshold that looks at total accuracy sees nothing. When the outage share rises to 0.18, the drop is 0.0475 and exceeds the run-to-run difference. That is, total accuracy is not a quality indicator but an indicator that depends on the outage share; it reports quality only when the outage becomes large enough.

The first row is the extreme case of this. Even when there is no outage at all, twelve requests are answered by the fallback, because the rate limit and the timeout produce a share in every run. The accuracy of these twelve requests is 0.7500; in normal operation, without any event record being written, 0.0300 of the requests are answered by a worse model.

The primary accuracy column stays in place in all four rows: 0.9044 / 0.9037 / 0.9032 / 0.8927. The primary model did not degrade; what degraded is the mix. The accuracy of the fallback tier, on the other hand, falls with the share (from 0.7500 to 0.5957), because as the outage grows, the proportion of long-context requests among those that fall to the fallback rises. The last column also writes the limit of this reading: the 0.6667 in the 0.06 row was computed from only twenty-four requests. It is a number with a small sample and cannot be tied to a threshold on its own; but it is far more informative than total accuracy, because it measures what it wants to measure.

Summary

• The fallback tier is not another server doing the same job: its window is narrow, its cue table is incomplete, and the label set does not narrow its candidate set.
• During the outage the fallback brings errors down from 26 to 2, raises total accuracy from 0.8450 to 0.8850 and adds 2227 tokens; against the no-outage baseline, it brings accuracy down from 0.8975 to 0.8850.
• The drop is 0.0125 and the run-to-run difference is 0.0200: an indicator that looks at total accuracy never sees the outage.
• When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667, difference 0.2371; the cost of this field is zero tokens.
• Total accuracy is not a quality indicator but an indicator that depends on the outage share: the drop exceeds the run-to-run difference only when the outage share reaches 0.18.
• The accuracy of the primary tier does not move across the four shares (between 0.9044 and 0.8927); what degrades is the mix, and even with no outage 0.0300 of the requests are answered by the fallback.

Next Step

In this lesson the run-to-run difference was used as a threshold: every difference below 0.0200 was counted as unmeasured. This threshold itself was never questioned. Why do the same workload, the same prompt and the same code give different numbers in two runs, which item moves when it is released, and does the movement stop if everything is pinned? The next lesson counts the items of a model call one by one, releases each of them alone and measures how many responses change, and separates how many of these items are in our hands.