--- title: 'Cost and Latency Control' source: 'https://academia.sh/en/courses/ai-agents/cost-and-latency-control' course: 'AI Agents' language: en updated: '2026-10-21T05:02:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Runs the same task set with two model classes and three step budgets and computes the cost per step and the cost per correct answer separately.' - 'Derives the band of unit price ratios in which the two measures reverse the ranking.' - 'Writes latency as a class read from the step count of the longest run and shows that this class moves from run to run.' --- # Cost and Latency Control Reading the spending of an agent loop with two separate denominators: the single-step call without tools is the cheapest at 90.00 cost units per step, but because its success is 0.0000, its cost per correct answer is undefined. When the large and small model classes are run with three step budgets, the small class stays below the 118.30 of the large class at 79.25 per step, while at 488.0 per correct answer it rises above its 453.5. The band where the two rankings diverge is given by the unit price ratio: the break-even ratio is 0.8957 per step and 0.5576 per correct answer, and at every price ratio in between the two measures give opposite answers. Latency is written as a class and read from the longest run; raw time is not measured. The previous lesson paid for the trace in tokens: the full observation level stored 13623 tokens and, in return, read back the cause of the failed tasks. That cost is only one item of the spending the loop runs up. The real spending is in the loop itself: each step carries a prompt, the carried prompt grows with the previous observations, and the number of steps changes from task to task. The question of this lesson is this: **what do we divide by** when we say a setup is expensive or not. The cost lesson in the Building Applications with Model APIs course set up token accounting per request and showed there too that the ranking changes when the denominator changes; that procedure is not repeated here. Here the denominator is **the step** or **the correct answer**, the thing divided is the spending not of a single call but of **the whole loop**, and the things compared are the step budget and the model class. - **AQ41.** The environment, tools, task set, and agent loop are **fictional**; no real model endpoint or tool server is called. The seed is **20260218**, the second run **20260219**. - **AQ42. The baseline is the single-step call without tools**: the agent calls no tool and answers in a single step. Its token is the token of the first step of the loop (prompt 60, answer 30). Because no observation enters the state, its success is **0.0000**. - **AQ43. The model class** scales the share of selecting the right tool. The multiplier of the **large class** is 1.00, that is, the shared definition itself; the multiplier of the **small class** is **0.60**, and the share is clipped between 0.20 and 0.95. The classes are referred to by type name; no model, version, or provider name appears. - **AQ44. The cost unit** is the relative price paid per token. It is taken as 1.00 in the large class; the unit price of the small class is a **parameter** and is printed in the table as **0.60**. - **AQ45. Latency is written as a class and is read from the step count of the longest run**: 1 step **single call**, 2 to 4 steps **short chain**, 5 to 8 steps **long chain**, nine and above **very long chain**. Raw time is not measured anywhere; what determines the class is the length of the chain in the worst case. - **AQ46. The break-even ratio** is the value of the unit price of the small class at which the two classes tie on the measure in question. It is derived independently of the chosen price and is read from the table itself. ## Six Setups, One Task Set The block below sets up the shared definition of the course: the fictional subscriber records of the measurement network, the fictional document set, five tools, twenty-four tasks, and the agent loop. ```python # FICTIONAL. The measurement network's subscriber records, document set, five tools and # twenty-four tasks are defined within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} ``` The model class touches the shared definition at a single place: a multiplier that scales the share of selecting the right tool. With the multiplier at 1.00 the baseline is kept exactly, that is, the large class is the shared definition itself. ```python # The model class scales the share of selecting the right tool; the large class is the shared definition itself. MULTIPLIER = [1.00] _base_selection = selection def selection(d): return max(0.20, min(0.95, _base_selection(d) * MULTIPLIER[0])) MODEL_CLASS = {"large": 1.00, "small": 0.60} # multiplier of the share of selecting the right tool UNIT_PRICE = {"large": 1.00, "small": 0.60} # cost unit per token NO_TOOL_TOKENS = 60 + 30 # token of the single-step call without tools def latency(longest): if longest <= 1: return "single call" return "short chain" if longest <= 4 else ("long chain" if longest <= 8 else "very long chain") def run(cls, budget, seed): MULTIPLIER[0] = MODEL_CLASS[cls] correct, steps, tok, longest = 0, 0, 0, 0 for g in TASKS: s = agent(g, max_steps=budget, seed=seed) steps += s["steps"] tok += s["tokens"] longest = max(longest, s["steps"]) correct += s["answer"] == correct_answer(g) return correct, steps, tok, longest for seed in (20260218, 20260219): n = len(TASKS) print(f"seed {seed}") print("setup success steps tokens cost per step per correct latency") print(f"{'no tools':<15} {0.0:>8.4f} {n:>6} {n * NO_TOOL_TOKENS:>9} {n * NO_TOOL_TOKENS:>8.1f}" f" {NO_TOOL_TOKENS:>12.2f} {'undefined':>13} {latency(1)}") data = {} for cls in ("large", "small"): for budget in (4, 6, 12): correct, steps, tok, longest = run(cls, budget, seed) data[(cls, budget)] = (correct, steps, tok) cost = tok * UNIT_PRICE[cls] print(f"{cls + ' budget ' + str(budget):<15} {correct / n:>8.4f} {steps:>6} {tok:>9} {cost:>8.1f}" f" {cost / steps:>12.2f} {cost / correct:>13.1f} {latency(longest)}") print("budget break-even ratio per step per correct answer") for budget in (4, 6, 12): lc, ls, lt = data[("large", budget)] sc, ss, st = data[("small", budget)] print(f"{budget:>6} {(lt / ls) / (st / ss):>28.4f} {(lt / lc) / (st / sc):>20.4f}") ``` ``` seed 20260218 setup success steps tokens cost per step per correct latency no tools 0.0000 24 2160 2160.0 90.00 undefined single call large budget 4 0.8750 84 9402 9402.0 111.93 447.7 short chain large budget 6 1.0000 92 10884 10884.0 118.30 453.5 long chain large budget 12 0.9583 98 12214 12214.0 124.63 531.0 very long chain small budget 4 0.5417 90 10624 6374.4 70.83 490.3 short chain small budget 6 0.7917 117 15454 9272.4 79.25 488.0 long chain small budget 12 1.0000 142 21762 13057.2 91.95 544.0 very long chain budget break-even ratio per step per correct answer 4 0.9482 0.5478 6 0.8957 0.5576 12 0.8132 0.5857 seed 20260219 setup success steps tokens cost per step per correct latency no tools 0.0000 24 2160 2160.0 90.00 undefined single call large budget 4 0.8333 83 9366 9366.0 112.84 468.3 short chain large budget 6 0.9583 96 11610 11610.0 120.94 504.8 long chain large budget 12 0.9583 98 11994 11994.0 122.39 521.5 long chain small budget 4 0.3750 89 10550 6330.0 71.12 703.3 short chain small budget 6 0.7917 121 16396 9837.6 81.30 517.8 long chain small budget 12 0.8750 140 20526 12315.6 87.97 586.5 very long chain budget break-even ratio per step per correct answer 4 0.9519 0.3995 6 0.8925 0.5850 12 0.8348 0.5335 ``` ## Cheap Relative to What The first row sets up the question by itself. The call without tools is cheaper than every setup of the large class at **90.00 per step**; it is below 111.93, 118.30, and 124.63. Its cost per correct answer is **undefined**, because the denominator is zero. At the very top of the cost-per-step ranking sits the setup that finishes no work. This is not an error of the measure but the result of what the denominator counts: cost per step measures **the cheapness of the step**; it does not measure whether the work got done. Within the same class, enlarging the budget steadily raises the cost per step: 111.93, 118.30, 124.63 in the large class. The reason is in the loop's token accounting; each step also carries the observations before it, so late steps are more expensive than early ones. The cost per correct answer, by contrast, is 447.7, 453.5, and 531.0. The difference of 5.8 between the first two values is smaller than the difference between the two runs of the same row (447.7 against 468.3, that is, 20.6); the ranking between budget 4 and budget 6 **counts as unmeasured** on this measure. The 531.0 of budget 12 is large relative to the band and is a real deterioration. ## The Band Where the Two Rankings Part The main divergence is between the classes. With the unit price taken as 0.60, the small class at budget 6 is clearly below the large class's **118.30** per step, at **79.25**. Read per correct answer, the same two rows reverse their order: **488.0** against **453.5**. The small class produces cheap steps but spends more of them (117 against 92) and finishes fewer tasks (0.7917 against 1.0000). The second run gives the same direction: 81.30 against 120.94 and 517.8 against 504.8. This divergence may look dependent on the chosen price, but that it is not can be read from the last section of the table. At budget 6 the break-even ratio per step is **0.8957** and the break-even ratio per correct answer is **0.5576**. If the unit price of the small class is below 0.5576, both measures count the small class as cheap; if it is above 0.8957, both measures count the large class as cheap. **At every price ratio in between, the two measures give opposite answers**, and the question of which one is cheap has no single right answer. The second run puts the band between 0.5850 and 0.8925; the upper edges of the two runs overlap by 0.0032, and the lower edge moves by 0.0274. For this reason the band is roughly between **0.56 and 0.89**. The lower edge at budget 4, however, moves between 0.5478 and 0.3995; the band of that row **counts as unmeasured**. The latency column carries a separate warning. The class is read from the longest run, and the large class at budget 12 comes out **very long chain** in one run and **long chain** in the other. The mean step count is almost the same in the two runs; what moves is the worst case of a single task. If a latency budget is set by the worst case, the class of the same setup can change between two runs, and this change cannot be read as an improvement or a deterioration. ## Summary - The single-step call without tools is the cheapest setup at 90.00 per step and its cost per correct answer is undefined; the per-step measure does not count whether the work got done. - In the large class, as the budget grows, the cost per step rises from 111.93 to 124.63, because late steps also carry the previous observations. - At a unit price of 0.60, the small class is cheap at 79.25 per step and expensive at 488.0 per correct answer; the counterparts of the large class are 118.30 and 453.5. - The break-even ratios give the band where the two measures part: at budget 6, 0.8957 per step and 0.5576 per correct answer; at every price in between, the two measures give opposite answers. - Latency is a class read from the longest run, and the same setup can fall into two different classes in two runs; raw time is not measured anywhere. ## Next Step One column stayed silent in all the rows of the cost table: the tool error. What the loop does when a tool returns an error was never asked in this lesson; the loop of the shared definition saw the error and turned toward the same goal again, and we had not written this down as a **policy**. The next lesson measures this gap: stopping at the first error, retrying the same tool, and never selecting the failed tool again are three separate policies, and the three pull the success, the steps, and the tool errors in separate directions. That lesson is also the close of the course.