--- title: 'Measuring Task Success and Step Efficiency Separately' source: 'https://academia.sh/en/courses/ai-agents/measuring-task-success-and-step-efficiency-separately' course: 'AI Agents' language: en updated: '2026-10-21T05:02:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the correctness of the tool sequence in the step trace as a column separate from the correctness of the answer.' - 'Counts the four combinations of a correct sequence and a correct answer and shows that a single success rate hides all four.' - 'Computes steps per correct answer and explains why this measure, read alone, favors the toolless baseline.' - 'Compares two runs of the same task set to determine which difference stays below the measurement band.' --- # Measuring Task Success and Step Efficiency Separately Showing on steps that a single success rate stacks four separate states: on a fictional set of twenty-four tasks the toolless single-step baseline comes out at 0.2917, and all of that rate sits in the wrong sequence, right answer box, because all seven document tasks are answered without any tool being called. The loop gives 0.8750 at the narrow budget and 0.9583 at the course budget; at the course budget the only failure is in the right sequence, wrong answer box and comes from a task whose sequence was completed in full being spoiled by an extra call in between. Steps per correct answer come out at 3.4286 for the baseline and 4.2174 at the course budget, so read alone, efficiency rewards the baseline that calls no tool; on the seventeen tasks that need tools the baseline gives zero correct answers. The run-to-run difference is 0.0000 at the baseline and 0.0417 at the narrow budget. Sixteen lessons built and equipped an agent: the loop was written, the place where it stops was settled, side-effect calls were put behind approval, tools were defined, executed, and limited in capability, memory was fitted into the context window, long-term records were written, compacted, and expired. One question was never asked: how do we know this agent works. What usually remains of an agent run is a single rate: how many tasks were answered correctly. The claim of this lesson is that the rate stacks **four separate states** on top of each other, and that from inside the stack you cannot read whether the agent found the answer **by doing the work or without doing it**. The same distinction was built in the Embeddings and Retrieval-Augmented Generation course on the right chunk and the right answer; the procedure stayed there and is not repeated here. What carries over is only **the distinction itself**, this time over **steps**. - **AQ1.** Subscriber records, the document set, five tools, twenty-four tasks, and the agent loop are **fictional** and are generated inside the lesson. No real model endpoint or tool server is **called**; tool selection comes from a single draw that depends on depth. - **AQ2.** The **baseline is a single-step call with no tools**; the simulator sees only the request text. - **AQ3.** **Sequence measure**: if the task's expected tool sequence appears in the step trace **in order and without errors**, the sequence is correct; an extra call in between does not break the sequence. - **AQ4.** **Answer measure**: if the result the loop leaves equals the correct answer exactly, the answer is correct. - **AQ5.** **Step efficiency** is steps per correct answer; the other units of cost are **tool calls and tokens**, and no duration is written. - **AQ6.** Every measure is printed over **two runs** (seeds 20260218 and 20260219). **A difference below the run-to-run difference counts as unmeasured.** ## A Fictional Environment, Five Tools, and the Agent Loop Five tools read a meter, compute consumption, apply a tariff, search documents, and write a record; the last one has a side effect. Each task is solved by a known sequence. ```python # FICTIONAL. Subscriber records, document set, tool set, task set and agent # loop are generated within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} KINDS = ("charge", "consumption", "document", "record") counts = {t: sum(1 for g in TASKS if g["kind"] == t) for t in KINDS} print(f"fictional environment {len(SUBSCRIBERS)} subscribers , {len(DOCUMENTS)} documents , {len(TOOLS)} tools") print("tasks", len(TASKS), ",", " , ".join(f"{k} {v}" for k, v in counts.items())) ``` ``` fictional environment 60 subscribers , 5 documents , 5 tools tasks 24 , charge 6 , consumption 8 , document 7 , record 3 ``` ## Sequence and Answer: Two Separate Columns The first column never looks at the answer: did the expected sequence appear in the step trace in order and without errors. The second column never looks at the trace. Where the two columns cross, every task falls into one of four boxes. ```python def toolless(g, seed=SEED): """Baseline: a single-step call with no tools. No tool is called; the simulator sees only the request text and guesses from the code prefix.""" r = rng(seed ^ trace_id("tek" + g["name"])) prefix = {"sayac": "SY", "tarif": "TR", "ariza": "AR", "kesin": "KS"} b = set(tokenize(g["request"])) for k, p in prefix.items(): if k in b: candidates = [c for c in DOCUMENTS if c.startswith(p)] return candidates[int(r() * len(candidates))] return 10 * int(r() * 20) def sequence_correct(g, trace): """Does the expected tool sequence appear in the step trace in order and without errors.""" k = 0 for name, failed in trace: if k < len(g["sequence"]) and name == g["sequence"][k] and not failed: k += 1 return k == len(g["sequence"]) def four_states(tasks, budget, seed=SEED): """Budget zero runs the baseline, otherwise the loop runs.""" d = {"cc": 0, "cw": 0, "wc": 0, "ww": 0, "steps": 0, "tok": 0, "calls": 0} c0 = CALLS[0] for g in tasks: s = agent(g, max_steps=budget, seed=seed) if budget else None trace, answer, steps, tok = ((s["trace"], s["answer"], s["steps"], s["tokens"]) if s else ([], toolless(g, seed), 1, 60)) sc, ac = sequence_correct(g, trace), answer == correct_answer(g) d["cc" if sc and ac else "cw" if sc else "wc" if ac else "ww"] += 1 d["steps"] += steps d["tok"] += tok d["calls"] = CALLS[0] - c0 return d SETUPS = (("toolless", 0), ("budget 4", 4), ("budget 8", 8)) print("setup c+c c+w w+c w+w success 2nd run diff steps calls tokens steps/correct") for name, b in SETUPS: d, e = four_states(TASKS, b), four_states(TASKS, b, 20260219) n, m = (d["cc"] + d["wc"]) / 24, (e["cc"] + e["wc"]) / 24 print(f"{name:<9} {d['cc']:>3} {d['cw']:>4} {d['wc']:>4} {d['ww']:>4} {n:>8.4f} {m:>8.4f}" f" {m - n:>+9.4f} {d['steps']:>6} {d['calls']:>6} {d['tok']:>9}" f" {d['steps'] / max(n * 24, 1):>14.4f}") print("\ntask kind breakdown (c+c / c+w / w+c / w+w)") print(f"{'kind':<11} {'tasks':>6} {'toolless':>16} {'budget 8':>16}") for kind in KINDS: gs = [g for g in TASKS if g["kind"] == kind] row = [" / ".join(str(four_states(gs, b)[k]) for k in ("cc", "cw", "wc", "ww")) for b in (0, 8)] print(f"{kind:<11} {len(gs):>6} {row[0]:>16} {row[1]:>16}") needs_tools = [g for g in TASKS if g["kind"] != "document"] print(f"\n{len(needs_tools)} tasks that need tools , correct answers and steps") for name, b in SETUPS: d = four_states(needs_tools, b) print(f"{name:<9} correct {d['cc'] + d['wc']:>3} / {len(needs_tools)} steps {d['steps']:>4}") ``` ``` setup c+c c+w w+c w+w success 2nd run diff steps calls tokens steps/correct toolless 0 0 7 17 0.2917 0.2917 +0.0000 24 0 1440 3.4286 budget 4 20 0 1 3 0.8750 0.8333 -0.0417 84 64 9402 4.0000 budget 8 23 1 0 0 0.9583 0.9583 +0.0000 97 74 11978 4.2174 task kind breakdown (c+c / c+w / w+c / w+w) kind tasks toolless budget 8 charge 6 0 / 0 / 0 / 6 6 / 0 / 0 / 0 consumption 8 0 / 0 / 0 / 8 8 / 0 / 0 / 0 document 7 0 / 0 / 7 / 0 7 / 0 / 0 / 0 record 3 0 / 0 / 0 / 3 2 / 1 / 0 / 0 17 tasks that need tools , correct answers and steps toolless correct 0 / 17 steps 17 budget 4 correct 14 / 17 steps 64 budget 8 correct 16 / 17 steps 77 ``` ## Reading the Four Combinations The first row gives the baseline. One toolless step scores **0.2917**, and **all of that rate sits in the third box**: wrong sequence, right answer. The calls column is zero; all seven document tasks came out right, because the request of these tasks is the same text and the word sayac (meter) inside it gives away the prefix of the document code. One number says the agent solved more than a quarter of the tasks; the four-state table says on the same row that **no work was done**. The real demonstration is in the last row. At the eight-step budget the only failure is **in the second box**: right sequence, wrong answer. In one record task the four expected tools passed in order and without errors, but an extra document search was made between the tariff and the record, and that call replaced the result in the state with a document code. The agent did the right work in the right order, then spoiled its own result. The table writes here **which fix is needed**, because a tool selection defect and a state corruption defect are repaired in different places. The narrow budget row shows that the third box can fill in the loop as well. In one record task the right amount was found, but the budget did not reach the record write: sequence incomplete, answer right. A single number counts this task as a success, yet the side-effect step never happened. The breakdown by kind collects all seven right answers of the baseline under the document kind, and the only failure of the eight-step budget under the record kind. The two runs say which difference can be counted. At the narrow budget the difference is **0.0417**, a single task; at the baseline and at the eight-step budget it is **0.0000**. The rise of 0.0833 between the two budgets is twice the band and can be counted; the size of the individual box transitions **counts as unmeasured**. ## Steps per Correct Answer The second measure ties cost to success. At the eight-step budget the agent spends 97 steps and produces 23 correct answers: **4.2174** per step. At the narrow budget it is 84 steps and 21 correct answers, that is, **4.0000**. The trap is in the first row. The toolless baseline gives 7 correct answers in 24 steps, and step efficiency comes out at **3.4286**, the best of the three setups. The setup that calls no tool and computes nothing wins on the efficiency measure, because this measure puts only the correct answers in the denominator and never asks **which tasks could be solved**. The last table fixes the denominator. Counting only the seventeen tasks that need tools, the baseline gives **0 / 17** and efficiency stays undefined on this subset; the narrow budget gives 14 and the eight-step budget 16 correct answers. This is why the measure is not read alone: unless **the task set and the success level** it was measured at are written down, the shortest way to make an agent cheaper is to reduce the tasks it can solve. ## Summary - A single success rate stacks four states: right sequence with right answer, right sequence with wrong answer, wrong sequence with right answer, wrong sequence with wrong answer. - The toolless baseline comes out at 0.2917, and all of that rate sits in the **wrong sequence, right answer** box; the agent called no tool. - At the eight-step budget the success is 0.9583 and the only failure is in the **right sequence, wrong answer** box: the sequence was completed in full, and an extra call in between spoiled the result. - Steps per correct answer are 3.4286 for the baseline and 4.2174 at the eight-step budget; read alone, the measure rewards the baseline, because on the seventeen tasks that need tools the baseline gives zero correct answers. - The run-to-run difference is 0.0000 at the baseline and at the eight-step budget, and 0.0417 at the narrow budget; box transitions of a single task stay below this band. ## Next Step This lesson split the number of an agent run in two: whether the work was done and whether the answer is correct. Both assume one thing: that the tools themselves are correct. If one of the tools computed wrongly, swallowed a missing field silently, or had a definition that did not match its function, the table would still fill in and would not say which tool is broken. The next lesson separates every tool from the loop and tests them one by one: how many of the defects placed in the five tools does this testing catch, and **which defect class never shows up in this layer**.