Skip to content
academia.sh
tr
Log in

Unit 01 of 03

Agent Architecture

Building the combination of model, tool and loop and counting where the loop stops: the success of a single-step tool-less call against the loop on the same task set, the share of picking the right tool falling as step depth grows and read separately across four task kinds, a single loop compared with a planner and executor split in success, steps and tokens, a step budget raised from two steps to twelve with success first rising and then falling and that fall not counting as a finding because it stays inside the two run bands, a self-critique round with the wrong answers it catches and the steps and tokens it adds, a second agent on the same task set with the success it adds, the tokens of the handed-off context and the number of fields lost in the handoff, and a side-effect tool with the records written when approval is on and off and the wrong writes stopped.

Published Updated

Lessons

  1. 01 What Is an Agent Building the combination of model, tool and loop and counting the whole loop instead of a single answer: on 24 fictional tasks a single-step tool-less call gives a task success of 0.0417 while a loop with a budget of eight steps gives 0.9583, and on the second run the baseline falls to 0.0000 while the loop stays at 0.9583. The cost is a rise from 24 steps to 97 steps, from zero tool calls to 74 calls, from 1440 tokens to 11978 tokens and from zero to 8 tool errors. Read by kind, the loop finishes every task of three steps or fewer and stays at two thirds on four-step write tasks: the place where success falls is step depth. The agent does not learn; the only thing that changes over a run is the observations it holds. 13 min read
  2. 02 The Agent Loop Opening one step of the loop into perception, reasoning, action and observation and counting the share of picking the right tool by step depth: on 24 fictional tasks the share is 0.8000 at depth 1 and 0.5000 at depth 4; on the second run 0.7742 and 0.5000. The fall of 0.3000 between them stands far above the widest run-to-run band (0.0644) and is measured, but the differences of 0.0917 and 0.0160 between neighboring depths stay inside the band and count as unmeasured. As depth grows the measurement itself thins out: the number of selections falls from 30 to 6. Read across four task kinds, the success of tasks of one, two and three steps is 1.0000 on both runs and 0.6667 on four-step write tasks; steps per task rise from 2.86 to 7.67. A wrong selection does not only spend a step, it can also overwrite the accumulated result. 11 min read
  3. 03 The Planner and Executor Split What separating the side that decides from the side that runs adds and what it costs: on 24 fictional tasks the single loop gives a task success of 0.9583, 97 steps, 74 tool calls and 11978 tokens; once planner and executor are separated, success becomes 0.8750, steps 118, calls 71 and tokens 10345. The difference in success equals the split arrangement's own run-to-run difference and counts as unmeasured. The extra steps are exactly one planning step per task. Tokens fall by more than thirteen percent on both runs, because the executor carries the plan, not the reasoning trace. 0.8868 of the plan's positions are right and eighteen of the twenty-four plans are right from start to finish; since a wrong position is executed blindly, correction starts only after the plan runs out. 14 min read
  4. 04 Stopping Criteria Writing where the loop stops as the rule of the course and sweeping the step budget over the values two, four, six, eight and twelve: on 24 fictional tasks task success rises from 0.3750 to 1.0000 and then flattens at 0.9583; on the second run it is 0.4167, 0.8333, 0.9583, 0.9583, 0.9583. The fall from the sixth step to the eighth is 0.0417 and the run-to-run difference at the sixth step is also 0.0417, so the fall is inside the measurement band and cannot be declared a finding; its mechanism is shown on the trace of a single task and its effect counts as unmeasured. When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214. Stopping by budget is not a stopping criterion but a safety limit: at the sixth step three tasks stop because the budget ran out. 14 min read
  5. 05 Self-Critique and Verification What it adds and what it costs for an agent to look at its own trace after it stops: when three observable signs are read on twenty-four fictional tasks, the round flags every wrong answer at narrow budgets but cannot see at all a wrong answer with a clean trace at eight steps. At a budget of four steps success rises from 21/24 to 23/24, in return for steps going from 84 to 152 and tokens from 9402 to 17052. When the same steps are given to a single longer run, the eight-step budget gives the same 23/24 with 97 steps and 11978 tokens. At six steps the round lowers success from 24/24 to 23/24; at eight steps it changes nothing, and six of the seven flags are unnecessary reruns. The run-to-run difference is one task, so the difference at six and eight steps counts as unmeasured. 13 min read
  6. 06 Multi-Agent Patterns Measuring a single agent and two agents side by side on the same task set: when the eight-step budget is divided in two and the job is distributed between a reader and a calculator, overall success stays at 23/24 on both runs, but steps rise from 97 to 140 and the handed-over context carries 363 tokens. The split raises the four-tool record tasks from 2/3 to 3/3 while it lowers the two-tool consumption tasks from 8/8 to 7/8. When the handoff packet is made cheaper, 178 tokens are gained, 68 fields are lost in return and success falls from 23/24 to 18/24; because the lost field is the subscriber type, the charge of commercial subscribers comes out exactly half. When the four-step budget is divided in two, the arrangement falls from 21/24 to 19/24 and from 20/24 to 16/24. 13 min read
  7. 07 Keeping the Human in the Loop Measuring the approval point placed in front of the tool with a side effect: with approval off, five records are written in twenty-four fictional tasks and three are wrong, and on the second run eight records are written and six are wrong. When approval is opened only on the call with a side effect, the written records fall to three and the wrong records to one; the cost is seven and eleven questions. When approval is spread to every step, the blocked writes do not change at all, but the questions rise to 74 and 76 and the tokens from 12103 to 14448. Task success stays fixed at 23/24 in all three scopes, so success does not measure the side effect. The number of blocked calls is larger than the number of prevented records, because some of the blocked calls would have failed anyway with a missing field. 14 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close