Cost and Latency Control

The previous lesson paid for the trace in tokens: the full observation level stored 13623 tokens and, in return, read back the cause of the failed tasks. That cost is only one item of the spending the loop runs up. The real spending is in the loop itself: each step carries a prompt, the carried prompt grows with the previous observations, and the number of steps changes from task to task.

The question of this lesson is this: what do we divide by when we say a setup is expensive or not. The cost lesson in the Building Applications with Model APIs course set up token accounting per request and showed there too that the ranking changes when the denominator changes; that procedure is not repeated here. Here the denominator is the step or the correct answer, the thing divided is the spending not of a single call but of the whole loop, and the things compared are the step budget and the model class.

• AQ41. The environment, tools, task set, and agent loop are fictional; no real model endpoint or tool server is called. The seed is 20260218, the second run 20260219.
• AQ42. The baseline is the single-step call without tools: the agent calls no tool and answers in a single step. Its token is the token of the first step of the loop (prompt 60, answer 30). Because no observation enters the state, its success is 0.0000.
• AQ43. The model class scales the share of selecting the right tool. The multiplier of the large class is 1.00, that is, the shared definition itself; the multiplier of the small class is 0.60, and the share is clipped between 0.20 and 0.95. The classes are referred to by type name; no model, version, or provider name appears.
• AQ44. The cost unit is the relative price paid per token. It is taken as 1.00 in the large class; the unit price of the small class is a parameter and is printed in the table as 0.60.
• AQ45. Latency is written as a class and is read from the step count of the longest run: 1 step single call, 2 to 4 steps short chain, 5 to 8 steps long chain, nine and above very long chain. Raw time is not measured anywhere; what determines the class is the length of the chain in the worst case.
• AQ46. The break-even ratio is the value of the unit price of the small class at which the two classes tie on the measure in question. It is derived independently of the chosen price and is read from the table itself.

Six Setups, One Task Set

The block below sets up the shared definition of the course: the fictional subscriber records of the measurement network, the fictional document set, five tools, twenty-four tasks, and the agent loop.

# FICTIONAL. The measurement network's subscriber records, document set, five tools and
# twenty-four tasks are defined within the lesson; no real endpoint or tool server is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}

The model class touches the shared definition at a single place: a multiplier that scales the share of selecting the right tool. With the multiplier at 1.00 the baseline is kept exactly, that is, the large class is the shared definition itself.

# The model class scales the share of selecting the right tool; the large class is the shared definition itself.
MULTIPLIER = [1.00]
_base_selection = selection


def selection(d):
    return max(0.20, min(0.95, _base_selection(d) * MULTIPLIER[0]))


MODEL_CLASS = {"large": 1.00, "small": 0.60}   # multiplier of the share of selecting the right tool
UNIT_PRICE = {"large": 1.00, "small": 0.60}    # cost unit per token
NO_TOOL_TOKENS = 60 + 30                       # token of the single-step call without tools


def latency(longest):
    if longest <= 1:
        return "single call"
    return "short chain" if longest <= 4 else ("long chain" if longest <= 8 else "very long chain")


def run(cls, budget, seed):
    MULTIPLIER[0] = MODEL_CLASS[cls]
    correct, steps, tok, longest = 0, 0, 0, 0
    for g in TASKS:
        s = agent(g, max_steps=budget, seed=seed)
        steps += s["steps"]
        tok += s["tokens"]
        longest = max(longest, s["steps"])
        correct += s["answer"] == correct_answer(g)
    return correct, steps, tok, longest


for seed in (20260218, 20260219):
    n = len(TASKS)
    print(f"seed {seed}")
    print("setup            success  steps    tokens     cost     per step   per correct  latency")
    print(f"{'no tools':<15} {0.0:>8.4f} {n:>6} {n * NO_TOOL_TOKENS:>9} {n * NO_TOOL_TOKENS:>8.1f}"
          f" {NO_TOOL_TOKENS:>12.2f} {'undefined':>13}  {latency(1)}")
    data = {}
    for cls in ("large", "small"):
        for budget in (4, 6, 12):
            correct, steps, tok, longest = run(cls, budget, seed)
            data[(cls, budget)] = (correct, steps, tok)
            cost = tok * UNIT_PRICE[cls]
            print(f"{cls + ' budget ' + str(budget):<15} {correct / n:>8.4f} {steps:>6} {tok:>9} {cost:>8.1f}"
                  f" {cost / steps:>12.2f} {cost / correct:>13.1f}  {latency(longest)}")
    print("budget    break-even ratio per step   per correct answer")
    for budget in (4, 6, 12):
        lc, ls, lt = data[("large", budget)]
        sc, ss, st = data[("small", budget)]
        print(f"{budget:>6} {(lt / ls) / (st / ss):>28.4f} {(lt / lc) / (st / sc):>20.4f}")

seed 20260218
setup            success  steps    tokens     cost     per step   per correct  latency
no tools          0.0000     24      2160   2160.0        90.00     undefined  single call
large budget 4    0.8750     84      9402   9402.0       111.93         447.7  short chain
large budget 6    1.0000     92     10884  10884.0       118.30         453.5  long chain
large budget 12   0.9583     98     12214  12214.0       124.63         531.0  very long chain
small budget 4    0.5417     90     10624   6374.4        70.83         490.3  short chain
small budget 6    0.7917    117     15454   9272.4        79.25         488.0  long chain
small budget 12   1.0000    142     21762  13057.2        91.95         544.0  very long chain
budget    break-even ratio per step   per correct answer
     4                       0.9482               0.5478
     6                       0.8957               0.5576
    12                       0.8132               0.5857
seed 20260219
setup            success  steps    tokens     cost     per step   per correct  latency
no tools          0.0000     24      2160   2160.0        90.00     undefined  single call
large budget 4    0.8333     83      9366   9366.0       112.84         468.3  short chain
large budget 6    0.9583     96     11610  11610.0       120.94         504.8  long chain
large budget 12   0.9583     98     11994  11994.0       122.39         521.5  long chain
small budget 4    0.3750     89     10550   6330.0        71.12         703.3  short chain
small budget 6    0.7917    121     16396   9837.6        81.30         517.8  long chain
small budget 12   0.8750    140     20526  12315.6        87.97         586.5  very long chain
budget    break-even ratio per step   per correct answer
     4                       0.9519               0.3995
     6                       0.8925               0.5850
    12                       0.8348               0.5335

Cheap Relative to What

The first row sets up the question by itself. The call without tools is cheaper than every setup of the large class at 90.00 per step; it is below 111.93, 118.30, and 124.63. Its cost per correct answer is undefined, because the denominator is zero. At the very top of the cost-per-step ranking sits the setup that finishes no work. This is not an error of the measure but the result of what the denominator counts: cost per step measures the cheapness of the step; it does not measure whether the work got done.

Within the same class, enlarging the budget steadily raises the cost per step: 111.93, 118.30, 124.63 in the large class. The reason is in the loop’s token accounting; each step also carries the observations before it, so late steps are more expensive than early ones. The cost per correct answer, by contrast, is 447.7, 453.5, and 531.0. The difference of 5.8 between the first two values is smaller than the difference between the two runs of the same row (447.7 against 468.3, that is, 20.6); the ranking between budget 4 and budget 6 counts as unmeasured on this measure. The 531.0 of budget 12 is large relative to the band and is a real deterioration.

The Band Where the Two Rankings Part

The main divergence is between the classes. With the unit price taken as 0.60, the small class at budget 6 is clearly below the large class’s 118.30 per step, at 79.25. Read per correct answer, the same two rows reverse their order: 488.0 against 453.5. The small class produces cheap steps but spends more of them (117 against 92) and finishes fewer tasks (0.7917 against 1.0000). The second run gives the same direction: 81.30 against 120.94 and 517.8 against 504.8.

This divergence may look dependent on the chosen price, but that it is not can be read from the last section of the table. At budget 6 the break-even ratio per step is 0.8957 and the break-even ratio per correct answer is 0.5576. If the unit price of the small class is below 0.5576, both measures count the small class as cheap; if it is above 0.8957, both measures count the large class as cheap. At every price ratio in between, the two measures give opposite answers, and the question of which one is cheap has no single right answer. The second run puts the band between 0.5850 and 0.8925; the upper edges of the two runs overlap by 0.0032, and the lower edge moves by 0.0274. For this reason the band is roughly between 0.56 and 0.89. The lower edge at budget 4, however, moves between 0.5478 and 0.3995; the band of that row counts as unmeasured.

The latency column carries a separate warning. The class is read from the longest run, and the large class at budget 12 comes out very long chain in one run and long chain in the other. The mean step count is almost the same in the two runs; what moves is the worst case of a single task. If a latency budget is set by the worst case, the class of the same setup can change between two runs, and this change cannot be read as an improvement or a deterioration.

Summary

• The single-step call without tools is the cheapest setup at 90.00 per step and its cost per correct answer is undefined; the per-step measure does not count whether the work got done.
• In the large class, as the budget grows, the cost per step rises from 111.93 to 124.63, because late steps also carry the previous observations.
• At a unit price of 0.60, the small class is cheap at 79.25 per step and expensive at 488.0 per correct answer; the counterparts of the large class are 118.30 and 453.5.
• The break-even ratios give the band where the two measures part: at budget 6, 0.8957 per step and 0.5576 per correct answer; at every price in between, the two measures give opposite answers.
• Latency is a class read from the longest run, and the same setup can fall into two different classes in two runs; raw time is not measured anywhere.

Next Step

One column stayed silent in all the rows of the cost table: the tool error. What the loop does when a tool returns an error was never asked in this lesson; the loop of the shared definition saw the error and turned toward the same goal again, and we had not written this down as a policy. The next lesson measures this gap: stopping at the first error, retrying the same tool, and never selecting the failed tool again are three separate policies, and the three pull the success, the steps, and the tool errors in separate directions. That lesson is also the close of the course.