Skip to content
academia.sh
tr
Log in

Unit 03 of 03

Agent Quality

Measuring how one knows that a built agent works: the four-state table of the right sequence and the right answer, with a right answer produced by a wrong sequence counted separately; step efficiency read as steps per correct answer; the share of defects placed in five tools that tool-level testing catches, and the class of defects that is never seen; what the end-to-end flow test catches of what the tool level misses, and the residual that neither of the two layers catches; at the result only, tool names, and full observation trace levels, the share of failed tasks whose cause can be found, together with the stored token; the point where cost per step and cost per correct answer split the ranking; and a comparison of the recovery policies of stopping, retrying the same tool, and skipping to another tool by task success, steps, and tool errors.

Published Updated

Lessons

  1. 01 Measuring Task Success and Step Efficiency Separately Showing on steps that a single success rate stacks four separate states: on a fictional set of twenty-four tasks the toolless single-step baseline comes out at 0.2917, and all of that rate sits in the wrong sequence, right answer box, because all seven document tasks are answered without any tool being called. The loop gives 0.8750 at the narrow budget and 0.9583 at the course budget; at the course budget the only failure is in the right sequence, wrong answer box and comes from a task whose sequence was completed in full being spoiled by an extra call in between. Steps per correct answer come out at 3.4286 for the baseline and 4.2174 at the course budget, so read alone, efficiency rewards the baseline that calls no tool; on the seventeen tasks that need tools the baseline gives zero correct answers. The run-to-run difference is 0.0000 at the baseline and 0.0417 at the narrow budget. 13 min read
  2. 02 Testing Each Tool Separately from the Loop Counting how many of eleven fictional defects placed in five tools the tool-level test catches: the eleven-case test suite fails on all four value defects and all three contract defects, but passes clean on all four definition defects. The overall catch rate is 0.6364, and by class 1.0000 and 1.0000 against 0.0000. The mechanism of the unseen class is written down: the test suite calls the tool's function directly, while the loop goes through the field list the tool declares, and a definition defect lives only in that list. The payoff of the layer is independence from the run: on the same task set the task the loop fails shifts from G23 to G15 when the seed changes, while the set of failing cases of the test suite is empty and identical in both runs. 6 min read
  3. 03 Scenario-Based End-to-End Validation Counting side by side the defect sets caught by two testing layers: the twenty-four-scenario end-to-end flow test catches three of the four definition defects that the tool-level test cannot see at all, and fails on five of the eleven defects in total. The tool-level layer catches seven, the intersection of the two layers is two, and the union is ten of eleven defects, that is 0.9091. The two separate causes of a miss are told apart by the defect activation count: D2, D4, and D11 never activate once during the loop; D5, D6, and D7 activate but the scenario result does not change. The residual defect D11 is a defect whose function is correct but whose definition declares a field it never uses, and no run can see it. On the defect-free baseline the failing scenario in the two runs is G23 and G15. 9 min read
  4. 04 Monitoring and Logging Measuring what the step trace of an agent loop running in production reads back: in a 240-run stream made of ten runs of twenty-four tasks, fifteen tasks end wrong, and their causes are twelve wrong tool selections and three cases of wandering. The trace that stores only the result gives 0.0000 of these causes, the trace that counts tool names gives 0.4000, and the trace that stores the full observation gives 1.0000; the stored token rises from 2738 to 5954 and 13623, while the token per cause found falls from 992.3 to 908.2. 0.8689 of the sixty-one tool errors occur in runs that end correctly, that is, the tool error counter is a symptom, not a cause. 10 min read
  5. 05 Cost and Latency Control Reading the spending of an agent loop with two separate denominators: the single-step call without tools is the cheapest at 90.00 cost units per step, but because its success is 0.0000, its cost per correct answer is undefined. When the large and small model classes are run with three step budgets, the small class stays below the 118.30 of the large class at 79.25 per step, while at 488.0 per correct answer it rises above its 453.5. The band where the two rankings diverge is given by the unit price ratio: the break-even ratio is 0.8957 per step and 0.5576 per correct answer, and at every price ratio in between the two measures give opposite answers. Latency is written as a class and read from the longest run; raw time is not measured. 7 min read
  6. 06 Error Recovery Measuring what a failed tool call does to the loop with three recovery policies: in a production stream of two hundred forty runs, stopping at the first error gives a task success of 0.7500, 727 steps, 53 tool errors, and 81996 tokens; retrying the same tool raises success to 0.9375 and moves the steps to 900, the tool errors to 61, and the tokens to 106352. Dropping the failed tool and moving on to another does the opposite of what is expected: tool errors rise to 64, steps to 907, and success falls to 0.8917. The step efficiency that stopping gains stays within the band of the two runs. The lesson is the close of the AI Agents course and collects the without-scaffolding and with-scaffolding numbers of twenty-two lessons in a single table. 12 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close