Measuring Task Success and Step Efficiency Separately

Sixteen lessons built and equipped an agent: the loop was written, the place where it stops was settled, side-effect calls were put behind approval, tools were defined, executed, and limited in capability, memory was fitted into the context window, long-term records were written, compacted, and expired. One question was never asked: how do we know this agent works.

What usually remains of an agent run is a single rate: how many tasks were answered correctly. The claim of this lesson is that the rate stacks four separate states on top of each other, and that from inside the stack you cannot read whether the agent found the answer by doing the work or without doing it. The same distinction was built in the Embeddings and Retrieval-Augmented Generation course on the right chunk and the right answer; the procedure stayed there and is not repeated here. What carries over is only the distinction itself, this time over steps.

• AQ1. Subscriber records, the document set, five tools, twenty-four tasks, and the agent loop are fictional and are generated inside the lesson. No real model endpoint or tool server is called; tool selection comes from a single draw that depends on depth.
• AQ2. The baseline is a single-step call with no tools; the simulator sees only the request text.
• AQ3. Sequence measure: if the task’s expected tool sequence appears in the step trace in order and without errors, the sequence is correct; an extra call in between does not break the sequence.
• AQ4. Answer measure: if the result the loop leaves equals the correct answer exactly, the answer is correct.
• AQ5. Step efficiency is steps per correct answer; the other units of cost are tool calls and tokens, and no duration is written.
• AQ6. Every measure is printed over two runs (seeds 20260218 and 20260219). A difference below the run-to-run difference counts as unmeasured.

A Fictional Environment, Five Tools, and the Agent Loop

Five tools read a meter, compute consumption, apply a tariff, search documents, and write a record; the last one has a side effect. Each task is solved by a known sequence.

# FICTIONAL. Subscriber records, document set, tool set, task set and agent
# loop are generated within the lesson; no real endpoint or tool server is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}


KINDS = ("charge", "consumption", "document", "record")
counts = {t: sum(1 for g in TASKS if g["kind"] == t) for t in KINDS}
print(f"fictional environment {len(SUBSCRIBERS)} subscribers , {len(DOCUMENTS)} documents , {len(TOOLS)} tools")
print("tasks", len(TASKS), ",", " , ".join(f"{k} {v}" for k, v in counts.items()))

fictional environment 60 subscribers , 5 documents , 5 tools
tasks 24 , charge 6 , consumption 8 , document 7 , record 3

Sequence and Answer: Two Separate Columns

The first column never looks at the answer: did the expected sequence appear in the step trace in order and without errors. The second column never looks at the trace. Where the two columns cross, every task falls into one of four boxes.

def toolless(g, seed=SEED):
    """Baseline: a single-step call with no tools. No tool is called;
    the simulator sees only the request text and guesses from the code prefix."""
    r = rng(seed ^ trace_id("tek" + g["name"]))
    prefix = {"sayac": "SY", "tarif": "TR", "ariza": "AR", "kesin": "KS"}
    b = set(tokenize(g["request"]))
    for k, p in prefix.items():
        if k in b:
            candidates = [c for c in DOCUMENTS if c.startswith(p)]
            return candidates[int(r() * len(candidates))]
    return 10 * int(r() * 20)


def sequence_correct(g, trace):
    """Does the expected tool sequence appear in the step trace in order and without errors."""
    k = 0
    for name, failed in trace:
        if k < len(g["sequence"]) and name == g["sequence"][k] and not failed:
            k += 1
    return k == len(g["sequence"])


def four_states(tasks, budget, seed=SEED):
    """Budget zero runs the baseline, otherwise the loop runs."""
    d = {"cc": 0, "cw": 0, "wc": 0, "ww": 0, "steps": 0, "tok": 0, "calls": 0}
    c0 = CALLS[0]
    for g in tasks:
        s = agent(g, max_steps=budget, seed=seed) if budget else None
        trace, answer, steps, tok = ((s["trace"], s["answer"], s["steps"], s["tokens"]) if s
                                     else ([], toolless(g, seed), 1, 60))
        sc, ac = sequence_correct(g, trace), answer == correct_answer(g)
        d["cc" if sc and ac else "cw" if sc else "wc" if ac else "ww"] += 1
        d["steps"] += steps
        d["tok"] += tok
    d["calls"] = CALLS[0] - c0
    return d


SETUPS = (("toolless", 0), ("budget 4", 4), ("budget 8", 8))
print("setup     c+c  c+w  w+c  w+w  success  2nd run      diff  steps  calls    tokens  steps/correct")
for name, b in SETUPS:
    d, e = four_states(TASKS, b), four_states(TASKS, b, 20260219)
    n, m = (d["cc"] + d["wc"]) / 24, (e["cc"] + e["wc"]) / 24
    print(f"{name:<9} {d['cc']:>3} {d['cw']:>4} {d['wc']:>4} {d['ww']:>4} {n:>8.4f} {m:>8.4f}"
          f" {m - n:>+9.4f} {d['steps']:>6} {d['calls']:>6} {d['tok']:>9}"
          f" {d['steps'] / max(n * 24, 1):>14.4f}")

print("\ntask kind breakdown (c+c / c+w / w+c / w+w)")
print(f"{'kind':<11} {'tasks':>6} {'toolless':>16} {'budget 8':>16}")
for kind in KINDS:
    gs = [g for g in TASKS if g["kind"] == kind]
    row = [" / ".join(str(four_states(gs, b)[k]) for k in ("cc", "cw", "wc", "ww"))
           for b in (0, 8)]
    print(f"{kind:<11} {len(gs):>6} {row[0]:>16} {row[1]:>16}")

needs_tools = [g for g in TASKS if g["kind"] != "document"]
print(f"\n{len(needs_tools)} tasks that need tools , correct answers and steps")
for name, b in SETUPS:
    d = four_states(needs_tools, b)
    print(f"{name:<9} correct {d['cc'] + d['wc']:>3} / {len(needs_tools)}   steps {d['steps']:>4}")

setup     c+c  c+w  w+c  w+w  success  2nd run      diff  steps  calls    tokens  steps/correct
toolless    0    0    7   17   0.2917   0.2917   +0.0000     24      0      1440         3.4286
budget 4   20    0    1    3   0.8750   0.8333   -0.0417     84     64      9402         4.0000
budget 8   23    1    0    0   0.9583   0.9583   +0.0000     97     74     11978         4.2174

task kind breakdown (c+c / c+w / w+c / w+w)
kind         tasks         toolless         budget 8
charge           6    0 / 0 / 0 / 6    6 / 0 / 0 / 0
consumption      8    0 / 0 / 0 / 8    8 / 0 / 0 / 0
document         7    0 / 0 / 7 / 0    7 / 0 / 0 / 0
record           3    0 / 0 / 0 / 3    2 / 1 / 0 / 0

17 tasks that need tools , correct answers and steps
toolless  correct   0 / 17   steps   17
budget 4  correct  14 / 17   steps   64
budget 8  correct  16 / 17   steps   77

Reading the Four Combinations

The first row gives the baseline. One toolless step scores 0.2917, and all of that rate sits in the third box: wrong sequence, right answer. The calls column is zero; all seven document tasks came out right, because the request of these tasks is the same text and the word sayac (meter) inside it gives away the prefix of the document code. One number says the agent solved more than a quarter of the tasks; the four-state table says on the same row that no work was done.

The real demonstration is in the last row. At the eight-step budget the only failure is in the second box: right sequence, wrong answer. In one record task the four expected tools passed in order and without errors, but an extra document search was made between the tariff and the record, and that call replaced the result in the state with a document code. The agent did the right work in the right order, then spoiled its own result. The table writes here which fix is needed, because a tool selection defect and a state corruption defect are repaired in different places.

The narrow budget row shows that the third box can fill in the loop as well. In one record task the right amount was found, but the budget did not reach the record write: sequence incomplete, answer right. A single number counts this task as a success, yet the side-effect step never happened. The breakdown by kind collects all seven right answers of the baseline under the document kind, and the only failure of the eight-step budget under the record kind.

The two runs say which difference can be counted. At the narrow budget the difference is 0.0417, a single task; at the baseline and at the eight-step budget it is 0.0000. The rise of 0.0833 between the two budgets is twice the band and can be counted; the size of the individual box transitions counts as unmeasured.

Steps per Correct Answer

The second measure ties cost to success. At the eight-step budget the agent spends 97 steps and produces 23 correct answers: 4.2174 per step. At the narrow budget it is 84 steps and 21 correct answers, that is, 4.0000.

The trap is in the first row. The toolless baseline gives 7 correct answers in 24 steps, and step efficiency comes out at 3.4286, the best of the three setups. The setup that calls no tool and computes nothing wins on the efficiency measure, because this measure puts only the correct answers in the denominator and never asks which tasks could be solved.

The last table fixes the denominator. Counting only the seventeen tasks that need tools, the baseline gives 0 / 17 and efficiency stays undefined on this subset; the narrow budget gives 14 and the eight-step budget 16 correct answers. This is why the measure is not read alone: unless the task set and the success level it was measured at are written down, the shortest way to make an agent cheaper is to reduce the tasks it can solve.

Summary

• A single success rate stacks four states: right sequence with right answer, right sequence with wrong answer, wrong sequence with right answer, wrong sequence with wrong answer.
• The toolless baseline comes out at 0.2917, and all of that rate sits in the wrong sequence, right answer box; the agent called no tool.
• At the eight-step budget the success is 0.9583 and the only failure is in the right sequence, wrong answer box: the sequence was completed in full, and an extra call in between spoiled the result.
• Steps per correct answer are 3.4286 for the baseline and 4.2174 at the eight-step budget; read alone, the measure rewards the baseline, because on the seventeen tasks that need tools the baseline gives zero correct answers.
• The run-to-run difference is 0.0000 at the baseline and at the eight-step budget, and 0.0417 at the narrow budget; box transitions of a single task stay below this band.

Next Step

This lesson split the number of an agent run in two: whether the work was done and whether the answer is correct. Both assume one thing: that the tools themselves are correct. If one of the tools computed wrongly, swallowed a missing field silently, or had a definition that did not match its function, the table would still fill in and would not say which tool is broken. The next lesson separates every tool from the loop and tests them one by one: how many of the defects placed in the five tools does this testing catch, and which defect class never shows up in this layer.