Cost Monitoring

The previous three lessons measured the network side of the call: the request that hit the rate limit was retried, the repeated context was cached, and the latency class was lowered by changing the context size and the setup. All three made something cheaper, and none of them asked how much cheaper. The latency class is the user’s measure; it is not the unit of the bill.

The unit of this lesson is the token. No currency is written, because price depends on the provider and on time; the token count, on the other hand, is a number the setup itself produces and it carries from run to run. Two things are to be measured: how many tokens a request costs, and how this number is distributed across requests. Without the second, the first misleads.

• PQ31. The code is a simulator; no real endpoint is called. The tickets, the conversations and the documents are fictional, the seed is 20260218.
• PQ32. The procedure of cost accounting is not repeated. Metric collection, the dashboard and unit cost allocation were measured in the Observability and Reliability course, in the DevOps and Platform Engineering curriculum and in the Cloud Governance, Security and Cost course. The only thing added here is the unit: the cost of a request is written in tokens.
• PQ33. In this lesson the workload is conversations, not single tickets. The state is carried on the client and resent with every request; the third request of a conversation also contains the text of the first request. This is the source of the tail.
• PQ34. Conversation lengths are drawn with weights from the set [1, 2, 3, 5, 8, 12]. Even the last request of the longest conversation stays below the context window, so the trimming decision does not come into play in this lesson; the cost of trimming was measured in the Introduction to AI Engineering course.
• PQ35. A request consists of one or several calls. The retry procedure is not repeated; the ledger records only how many calls were made and the input and output tokens of each call.
• PQ36. On retry the simulator’s seed is increased by one. Because the same prompt with the same seed gives the same response, a second attempt can give a different result only this way.
• PQ37. The cost of a request is input plus output tokens; the input of a failed call is counted too, because it was sent.

The Per-Request Ledger

Reporting the total tokens as a single number is a sum, not accounting. Accounting requires writing which request a cost belongs to. The ledger below keeps one row for every request: how many calls were made, how many tokens went in, how many tokens came out.

# SIMULATOR -- FICTIONAL. No real model endpoint is called; the tickets, conversations
# and documents are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per token
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                        "faturami odedim ama sistemde borc gorunuyor",
                        "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                         "planli kesinti duyurusu nereden takip edilir",
                         "aksamdan beri kesinti yasiyoruz"]}
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "odeme",
        "odedi": "odeme", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza",
        "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "yardi": "bilgi", "nered": "bilgi", "kayit": "sikayet", "bekle": "sikayet"}
FREE_FORM = ["talebiniz {k} konusunda kaydedildi ve ekibe iletildi",
             "{k} ile ilgili kaydiniz olusturuldu en kisa surede donulecek",
             "kaydiniz alindi ilgili birim sizinle iletisime gececek"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "reply": ""}
    if COUNTER[0] % 37 == 0:                        # the rate limit depends on the call count
        return {"status": "rate_limit", "input": len(tok), "output": 0, "reply": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "reply": ""}
    candidates = [k for k in CONCEPTS if k[:5] in tok]      # the label set narrows the candidates
    p = {k: 0.15 for k in (candidates if len(candidates) >= 3 else CONCEPTS)}
    for t in tok:
        if CUES.get(t) in p:
            p[CUES[t]] += 1.0
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        w = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    y = k if "tek" in tok else FREE_FORM[int(r() * 3)].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:                       # stopping condition
        out, y = out[:max_output], " ".join(out[:max_output])
    return {"status": "ok", "input": len(tok), "output": len(out), "reply": y}


def read_label(y):
    return next((s for s in y.split() if s in TOPICS), None)


INSTRUCTION = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , "
               "kesinti . Yanit olarak tek etiket yaz . Talep: ")
DOCUMENTS = ["TR1 tarife basamaklari abone tipine gore belirlenir ve yilda bir guncellenir",
             "FT2 fatura itirazi kayit acildiktan sonra on is gunu icinde sonuclanir",
             "SY3 sayac degisimi randevusu saha ekibinin bolge programina baglidir",
             "AR4 ariza kaydi acilan adreste ekip ayni gun icinde yonlendirilir",
             "KS5 planli kesinti duyurulari bir hafta once ilan panosuna asilir"]
CONVERSATIONS, i = [], 0
while sum(len(c["turns"]) for c in CONVERSATIONS) < 400:    # FICTIONAL workload: conversations
    r = rng(SEED + 900 + i)
    i += 1
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    u = [1, 2, 3, 5, 8, 12][pick(r(), [0.46, 0.24, 0.14, 0.09, 0.05, 0.02])]
    CONVERSATIONS.append({"id": i, "doc": pick(r(), [0.74, 0.16, 0.07, 0.03]),
                          "turns": [{"topic": k, "text": TEMPLATES[k][(i + t) % 3]}
                                    for t in range(u)]})
REQUESTS = [{"conv": c, "pos": t, "ticket": c["turns"][t]}
            for c in CONVERSATIONS for t in range(len(c["turns"]))][:400]


def context(x, mode="full"):
    t = x["conv"]["turns"][:x["pos"]]
    t = [] if mode == "stateless" else (t[-2:] if mode == "two turns" else t)
    if mode == "summary state":
        g = "".join(f"Onceki konu: {o['topic']} . " for o in t)
    else:
        g = "".join(f"Onceki: {o['text']} Yanit: {o['topic']} . " for o in t)
    return g + "".join(DOCUMENTS[(x["conv"]["id"] + j) % 5] + " . "
                       for j in range(x["conv"]["doc"]))


def ledger(mode="full", seed=SEED):
    """A separate row for every request: calls, input tokens, output tokens."""
    COUNTER[0] = 0
    d = []
    for x in REQUESTS:
        c = inp = out = 0
        prompt = INSTRUCTION + context(x, mode) + x["ticket"]["text"]
        for a in range(3):
            s = call(prompt, seed + a)              # the seed grows on retry
            c, inp, out = c + 1, inp + s["input"], out + s["output"]
            if s["status"] == "ok":
                break
        d.append({"total": inp + out, "calls": c, "output": out,
                  "correct": read_label(s["reply"]) == x["ticket"]["topic"]})
    return d


base = ledger()
t = sorted(z["total"] for z in base)
total = sum(t)
print("conversations", len(CONVERSATIONS), " requests", len(base), " calls",
      sum(z["calls"] for z in base), " total tokens", total)
print("mean %.1f  median %d  p75 %d  p90 %d  p99 %d  max %d"
      % (total / 400, t[200], t[300], t[360], t[396], t[-1]))
for p, d in ((1, 4), (5, 20), (10, 40)):
    print("top %-4s %5d tokens, %.4f of the total"
          % (f"{p}%", sum(t[-d:]), sum(t[-d:]) / total))
base2 = ledger(seed=SEED + 1)
t2 = sorted(z["total"] for z in base2)
print("run 2  mean %.1f  median %d  top 1%% share %.4f"
      % (sum(t2) / 400, t2[200], sum(t2[-4:]) / sum(t2)))
print("correct %.4f / %.4f" % (sum(z["correct"] for z in base) / 400,
                               sum(z["correct"] for z in base2) / 400))

conversations 154  requests 400  calls 414  total tokens 32642
mean 81.6  median 69  p75 100  p90 139  p99 224  max 303
top 1%     975 tokens, 0.0299 of the total
top 5%    3892 tokens, 0.1192 of the total
top 10%   6932 tokens, 0.2124 of the total
run 2  mean 83.0  median 69  top 1% share 0.0409
correct 0.9125 / 0.8925

The Tail the Mean Hides

A report reduced to a single number would have said: four hundred requests, thirty-two thousand six hundred forty-two tokens, 81.6 per request. A plan that looks at this number writes the cost of doubling the request count as double too, and is wrong.

The median is 69, that is, half of the requests are below 69. What pulls the mean up is the upper percentiles: the 75th percentile is 100, the 90th percentile 139, the 99th percentile 224 and the most expensive request 303 tokens. The most expensive request is more than four times the median. This is not a defect; it is the unavoidable result of long conversations. Because the state is carried on the client, the twelfth request of a conversation resends the eleven turns of that conversation too.

The weight of the tail is read as a share. The most expensive one percent, that is, four of the four hundred requests, spends 0.0299 of the total: three times its own share. The most expensive ten percent spends 0.2124, more than twice its share. In an investigation of a budget overrun, the place to look is these forty requests; making each of the remaining three hundred sixty cheaper by one token does not give the same gain.

The last two lines test the stability of the measure itself. In the second run the mean goes from 81.6 to 83.0, that is, it moves by seventeen in a thousand; the median is constant at 69. By contrast the share of the most expensive one percent goes from 0.0299 to 0.0409, that is, it moves by more than a third. Because the tail rests on a small number of requests, it is noisy. A direct consequence follows: the tail share is not read from a single run, it is accumulated and read over a window. That the mean is stable and the tail volatile is a reason not to trust the mean but to measure the tail for longer.

The Items of the Bill

Where the total tokens go is found by counting the pieces of text inside the request separately. The block below splits the same ledger into items and then runs the three policies that shorten the conversation state on the same workload.

items = {"instruction": 400 * len(tokenize(INSTRUCTION)),
         "conversation state": sum(len(tokenize(context(x)))
                                   - len(tokenize(context(x, "stateless")))
                                   for x in REQUESTS),
         "document": sum(len(tokenize(context(x, "stateless"))) for x in REQUESTS),
         "ticket text": sum(len(tokenize(x["ticket"]["text"])) for x in REQUESTS),
         "output": sum(z["output"] for z in base)}
items["retries"] = total - sum(items.values())
print(f"{'item':<18} {'tokens':>8}  {'share':>7}")
for a, v in items.items():
    print(f"{a:<18} {v:>8}  {v / total:>7.4f}")
print(f"\n{'policy':<13} {'total':>7} {'per request':>11} {'p99':>9} {'top 1% share':>13}"
      f" {'correct':>7} {'per correct':>11}")
for mode in ("full", "two turns", "summary state", "stateless"):
    d = ledger(mode)
    s = sorted(z["total"] for z in d)
    tt, n_correct = sum(s), sum(z["correct"] for z in d)
    print(f"{mode:<13} {tt:>7} {tt / 400:>11.1f} {s[396]:>9} {sum(s[-4:]) / tt:>13.4f}"
          f" {n_correct / 400:>7.4f} {tt / n_correct:>11.1f}")

item                 tokens    share
instruction           12800   0.3921
conversation state    11939   0.3658
document               2344   0.0718
ticket text            3885   0.1190
output                  625   0.0191
retries                1049   0.0321

policy          total per request       p99  top 1% share correct per correct
full            32642        81.6       224        0.0299  0.9125        89.4
two turns       27401        68.5       198        0.0314  0.9125        75.1
summary state   24355        60.9       120        0.0259  0.9075        67.1
stateless       20198        50.5        95        0.0276  0.7975        63.3

The first reading of the item table is this: only 0.1190 of this bill carries new information. The ticket text, that is, the sentence the subscriber wrote in that request, is three thousand eight hundred eighty-five tokens. The two largest of the remaining items are repeated text, the instruction at 0.3921 and the conversation state at 0.3658, and together they make up nearly three quarters of the total. The output tokens are 0.0191, that is, negligible; in this setup the model writes a single label, and the cost of what it writes is invisible next to the cost of what it reads.

This split also says which scaffolding touches where. The instruction item is identical in every request, and prompt caching targets it; the conversation state item grows from request to request, and shortening targets it. Without counting the two separately, it cannot be known what each scaffolding will gain.

Which Unit Is Used to Look

The policy table compares four setups on the same workload. A reading that looks at the per request column is clear: dropping the state entirely takes 81.6 down to 50.5, that is, it gains more than a third. The accuracy of the same row is 0.7975, which is 0.1150 lower than the 0.9125 of the full state. Since the run-to-run difference is 0.0200, this drop is real, and it means forty-six wrong labels in four hundred requests.

The last column reads the same four rows per correct answer and narrows the difference: 89.4 / 75.1 / 67.1 / 63.3. The stateless setup, although its accuracy is the lowest, is the cheapest per correct answer. The two units point to the same winner but write the difference at a different size, because the first counts the cost of a wrong answer as zero. On a support line a wrong label is not free; forty-six wrongly routed tickets create manual correction work. Which unit is right is a decision of the product, not of the measurement, but writing which unit was used is the responsibility of the measurement.

The third row is a gain that cannot be measured. The summary state setup lowers the accuracy from 0.9125 to 0.9075; the difference is 0.0050, a quarter of the run-to-run difference. This row can be read as “it does not lower the accuracy”, but the correct reading is “whether it lowers it cannot be known with this measurement”. The 8287 tokens it gains in return, on the other hand, are measured.

The tail column also shows a reverse behavior. Although the two turns setup lowers the total from 32642 to 27401, the share of the most expensive one percent rises from 0.0299 to 0.0314. The share grows because shortening cuts the total more than it cuts the tail. Since the tail share is a ratio, it cannot be read alone; the total must be written next to it.

Summary

• The unit of cost is the token; no currency is written, because price depends on the provider, while the token count is a number the setup itself produces.
• The mean is 81.6, the median 69, the 99th percentile 224 and the largest request 303 tokens; a plan that looks at the mean cannot see the cost of long conversations.
• The most expensive one percent spends 0.0299 of the total, the most expensive ten percent 0.2124: a budget investigation is done by looking at forty requests.
• The mean moves from 81.6 to 83.0 across two runs while the tail share goes from 0.0299 to 0.0409; the tail is read not from a single run but from a window.
• Of the bill, 0.3921 is the instruction, 0.3658 is the conversation state, and only 0.1190 is the ticket text that carries new information; the output tokens are 0.0191.
• The same four policies rank the same in both units, but the difference narrows (50.5 against 63.3); without writing which unit was used, this difference cannot be read.

Next Step

The ledger of this lesson counts every request the same way: a request went, a response came, this many tokens were held. There is one column the ledger cannot see. When part of the four hundred requests pass through another route because the endpoint did not answer, they will write the same row too: the call count, the tokens, even a label. Where the response came from is not written to the ledger. The next lesson counts the share of requests that fall onto a second route in the face of an outage and prints the accuracy of those requests separately: once the outage is covered, the error visible to the user drops to almost zero, the total accuracy hardly moves, and the drop shows only when the source of the response is written to the ledger.