Skip to content
academia.sh
tr
Log in

Course Advanced

AI Agents

Published Updated

By the end of this course

Start course

01

Agent Architecture

Building the combination of model, tool and loop and counting where the loop stops: the success of a single-step tool-less call against the loop on the same task set, the share of picking the right tool falling as step depth grows and read separately across four task kinds, a single loop compared with a planner and executor split in success, steps and tokens, a step budget raised from two steps to twelve with success first rising and then falling and that fall not counting as a finding because it stays inside the two run bands, a self-critique round with the wrong answers it catches and the steps and tokens it adds, a second agent on the same task set with the success it adds, the tokens of the handed-off context and the number of fields lost in the handoff, and a side-effect tool with the records written when approval is on and off and the wrong writes stopped.

  1. 01 What Is an Agent Building the combination of model, tool and loop and counting the whole loop instead of a single answer: on 24 fictional tasks a single-step tool-less call gives a task success of 0.0417 while a loop with a budget of eight steps gives 0.9583, and on the second run the baseline falls to 0.0000 while the loop stays at 0.9583. The cost is a rise from 24 steps to 97 steps, from zero tool calls to 74 calls, from 1440 tokens to 11978 tokens and from zero to 8 tool errors. Read by kind, the loop finishes every task of three steps or fewer and stays at two thirds on four-step write tasks: the place where success falls is step depth. The agent does not learn; the only thing that changes over a run is the observations it holds.
  2. 02 The Agent Loop Opening one step of the loop into perception, reasoning, action and observation and counting the share of picking the right tool by step depth: on 24 fictional tasks the share is 0.8000 at depth 1 and 0.5000 at depth 4; on the second run 0.7742 and 0.5000. The fall of 0.3000 between them stands far above the widest run-to-run band (0.0644) and is measured, but the differences of 0.0917 and 0.0160 between neighboring depths stay inside the band and count as unmeasured. As depth grows the measurement itself thins out: the number of selections falls from 30 to 6. Read across four task kinds, the success of tasks of one, two and three steps is 1.0000 on both runs and 0.6667 on four-step write tasks; steps per task rise from 2.86 to 7.67. A wrong selection does not only spend a step, it can also overwrite the accumulated result.
  3. 03 The Planner and Executor Split What separating the side that decides from the side that runs adds and what it costs: on 24 fictional tasks the single loop gives a task success of 0.9583, 97 steps, 74 tool calls and 11978 tokens; once planner and executor are separated, success becomes 0.8750, steps 118, calls 71 and tokens 10345. The difference in success equals the split arrangement's own run-to-run difference and counts as unmeasured. The extra steps are exactly one planning step per task. Tokens fall by more than thirteen percent on both runs, because the executor carries the plan, not the reasoning trace. 0.8868 of the plan's positions are right and eighteen of the twenty-four plans are right from start to finish; since a wrong position is executed blindly, correction starts only after the plan runs out.
  4. 04 Stopping Criteria Writing where the loop stops as the rule of the course and sweeping the step budget over the values two, four, six, eight and twelve: on 24 fictional tasks task success rises from 0.3750 to 1.0000 and then flattens at 0.9583; on the second run it is 0.4167, 0.8333, 0.9583, 0.9583, 0.9583. The fall from the sixth step to the eighth is 0.0417 and the run-to-run difference at the sixth step is also 0.0417, so the fall is inside the measurement band and cannot be declared a finding; its mechanism is shown on the trace of a single task and its effect counts as unmeasured. When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214. Stopping by budget is not a stopping criterion but a safety limit: at the sixth step three tasks stop because the budget ran out.
  5. 05 Self-Critique and Verification What it adds and what it costs for an agent to look at its own trace after it stops: when three observable signs are read on twenty-four fictional tasks, the round flags every wrong answer at narrow budgets but cannot see at all a wrong answer with a clean trace at eight steps. At a budget of four steps success rises from 21/24 to 23/24, in return for steps going from 84 to 152 and tokens from 9402 to 17052. When the same steps are given to a single longer run, the eight-step budget gives the same 23/24 with 97 steps and 11978 tokens. At six steps the round lowers success from 24/24 to 23/24; at eight steps it changes nothing, and six of the seven flags are unnecessary reruns. The run-to-run difference is one task, so the difference at six and eight steps counts as unmeasured.
  6. 06 Multi-Agent Patterns Measuring a single agent and two agents side by side on the same task set: when the eight-step budget is divided in two and the job is distributed between a reader and a calculator, overall success stays at 23/24 on both runs, but steps rise from 97 to 140 and the handed-over context carries 363 tokens. The split raises the four-tool record tasks from 2/3 to 3/3 while it lowers the two-tool consumption tasks from 8/8 to 7/8. When the handoff packet is made cheaper, 178 tokens are gained, 68 fields are lost in return and success falls from 23/24 to 18/24; because the lost field is the subscriber type, the charge of commercial subscribers comes out exactly half. When the four-step budget is divided in two, the arrangement falls from 21/24 to 19/24 and from 20/24 to 16/24.
  7. 07 Keeping the Human in the Loop Measuring the approval point placed in front of the tool with a side effect: with approval off, five records are written in twenty-four fictional tasks and three are wrong, and on the second run eight records are written and six are wrong. When approval is opened only on the call with a side effect, the written records fall to three and the wrong records to one; the cost is seven and eleven questions. When approval is spread to every step, the blocked writes do not change at all, but the questions rise to 74 and 76 and the tokens from 12103 to 14448. Task success stays fixed at 23/24 in all three scopes, so success does not measure the side effect. The number of blocked calls is larger than the number of prevented records, because some of the blocked calls would have failed anyway with a missing field.

02

Tools and Memory

Counting one by one the surfaces the agent touches and the state it carries: the valid call rate and tool errors across the four levels of a tool definition (name only, name and description, schema, schema and example call); the effect of giving a result raw, truncated, or summarized on success and on accumulated tokens; the number of calls that exceed the declared resource limit, with the share of tasks the limit stops and the share that stopping lowers; completed tasks and stopped calls under full, read-only, and allowlisted permission sets; the share of correct answers and the number of steps carrying the wrong source in a mix of three source classes; the tokens carried per step as the loop grows, the steps that fit in the window, and the information that drops when the window fills; the records stored under the every-step, result-only, and selective write policies and the steps gained in later tasks; the tokens that fall as the compaction ratio rises and the number of tasks that need the discarded field; the stale records that fall and the valid-record loss that rises as the expiry threshold tightens.

  1. 01 Tool Definition Measuring what a tool's name, description, and schema add to the loop: on 24 fictional tasks the definition is given at four levels, and when only the name is given, 113 of 157 calls come back as tool errors, the valid call rate stays at 0.2803, and task success is 0.3333. Adding the description brings the errors down to 56 and raises task success to 0.6667. Adding the schema drops the call count to 74 and the errors to 8, and the valid rate becomes 0.8919, but task success stalls at 0.7917: the schema gives the field names, not which field carries which value, and two swapped indexes return 99948 instead of 52 in a call that passes the schema. The example call zeroes this share, raises task success to 0.9583, and changes the steps, the calls, and the tool errors not at all; its cost is the catalog growing from 97 to 164 tokens, and tokens per correct answer rising from 1126 to 1212.
  2. 02 Executing Tool Calls Measuring in which form the result a tool returns is given back to the model: on 24 fictional tasks the raw observation gives a task success of 0.9583 with 770 observation tokens and 10103 total tokens. Truncating the observation to seven words drops the tokens to 9160, but because it drops the subscriber type, four commercial invoices come out at exactly half and task success falls to 0.7917. Truncating to four words breaks the chain: tool errors jump from 8 to 116, steps from 97 to 156, total tokens to 15945, and task success stays at 0.2917; making it cheaper makes it more expensive. The summary that declares the fields to carry keeps the task success of the raw observation in both runs while dropping the observation tokens to 401 and the tokens per correct answer from 439 to 401. When the subscriber type is removed from the same summary, task success falls to the truncation level, 0.7917: a summary is only as good as its declaration.
  3. 03 Code Execution Tools Measuring, through its resource limits, a tool that runs the program an agent wrote inside a sandbox: three ways of writing a program that computes the same amount spend 15, 172, and 19 steps, hold 2 to 5 cells, and produce 1 to 5 outputs. At the loose limit no call exceeds the declared limit, but three tasks pass the 96-step task budget while the per-call limit stops nothing. The medium limit stops 24 of 28 code calls, and the classes it stops are 20 for steps and 4 for cells; at the tight limit the same 24 calls stop this time on 20 step and 4 output classes. Stopping does not drop a task directly; it drops it by consuming the step budget: on code tasks task success falls from 0.7778 to 0.3333, and in the second run from 1.0000 to 0.5556. The tight limit gives the same task success as the medium limit while cutting wasted steps from 1356 to 552. Running code is entirely on the defensive side.
  4. 04 File System and Terminal Access Counting permission restriction on the agent's file and command surface: twenty-four fictional surface tasks are run under three permission sets, and completed tasks and stopped calls are printed side by side. Under full permission 0.9583 of the tasks are completed and the gate stops no call; the read-only permission stops 71 calls but drops completed tasks to 0.5000; the allowlisted set stops 20 calls and stays at 0.8333, and all of its stops are gathered in a single task kind. Under read-only the correct answer is 0.6250 while completed tasks are 0.5000: in three tasks the answer was produced correctly but the write never happened. A stopped call also consumes a step; steps rise from 78 to 129. Although the totals stay the same in the two runs, two tasks swap places at the task level, so the measurement band is 0.0833 and the 0.1250 difference between full permission and the allowlist is above the band.
  5. 05 Web Search and Browsing Counting access to outside information as a source classification: eighteen fictional sources are classified as primary, secondary, and unverified, and twenty-four outside-information tasks are run in five setups. The loop without outside information gives 0.2083 correct answers; the primary source alone raises it to 0.7500; adding the secondary class to the mix drops it to 0.3750 and pollutes ten tasks; the three classes together raise the polluted tasks to fifteen. On the same pool, ranking the choice by class brings the correct answer back to 0.7500, lowers the polluted tasks to five, and leaves the call count unchanged. After a wrong code enters the state it is carried for an average of 3.40 to 4.00 steps, and in all of the polluted tasks it reaches the record-writing step. The measurement band is 0.0833, so both the 0.3750 drop and the 0.3750 recovery are measured.
  6. 06 Short-Term Memory Counting step by step the conversation state that the agent loop carries: in a single task the tokens carried are 79 at the first step and rise to 255 at the seventh, and most of the growth comes not from the state but from the trace (at the seventh step the state is 63 tokens and the trace load is 192). Twenty-four tasks are run at five window values. With an unlimited window the correct answer is 0.9583, the total is 9316 tokens, and tool errors are 8. At a window of 200 the correct answer falls to 0.8750, at 150 to 0.8333, at 120 to 0.7083; the dropped fields rise from 0 to 85 and tool errors from 8 to 50. The fields dropped most are the identity fields of the task: abone 16, anahtar 16, sutun 15. Narrowing the window does not lower the tokens carried, it raises them: because steps rise from 97 to 112, the total climbs from 9316 to 12026. The measurement band is 0.0833, so the 0.0833 drop at the window of 200 does not count as a finding.
  7. 07 Long-Term Memory Measuring three write policies on the same second set of 48 tasks: making the observation of every step a separate record accumulates 74 records and saves 44 steps, writing a single answer record per task saves 199 steps with 24 records but drops task success from 0.9167 to 0.5833, and the selective policy, which merges the observations in the plan into a single record, saves 118 steps, again with 24 records. What pays off is not the number of records but the unit of the record.
  8. 08 Context Compression Sweeping two compaction policies over the same 24 tasks: when the carried state is paid for as well, the baseline gives 21913 tokens and 0.9583 task success. When selective recall cuts the kept fields to three, tokens fall to 16569 and task success does not change at all, but the number of tasks that need the dropped field rises from 0 to 8. Summarization falls to 0.7917 on the same budget and at the tightest setting raises tokens to 33566, 1.53 times the baseline. This is the point where compaction makes the job worse.
  9. 09 Memory Decay Sweeping the expiry threshold over seven settings: in memory without a threshold 0.4528 of the reads are stale and task success is 0.6389. As the threshold tightens, the stale share falls to 0.0000 and task success rises to 0.9306, but the share of dropped records that are still valid climbs from 0.0000 to 0.9701 and steps rise from 138 to 315. The task success gained by tightening the threshold below eight stays as large as the run-to-run difference itself, that is, it counts as unmeasured.

03

Agent Quality

Measuring how one knows that a built agent works: the four-state table of the right sequence and the right answer, with a right answer produced by a wrong sequence counted separately; step efficiency read as steps per correct answer; the share of defects placed in five tools that tool-level testing catches, and the class of defects that is never seen; what the end-to-end flow test catches of what the tool level misses, and the residual that neither of the two layers catches; at the result only, tool names, and full observation trace levels, the share of failed tasks whose cause can be found, together with the stored token; the point where cost per step and cost per correct answer split the ranking; and a comparison of the recovery policies of stopping, retrying the same tool, and skipping to another tool by task success, steps, and tool errors.

  1. 01 Measuring Task Success and Step Efficiency Separately Showing on steps that a single success rate stacks four separate states: on a fictional set of twenty-four tasks the toolless single-step baseline comes out at 0.2917, and all of that rate sits in the wrong sequence, right answer box, because all seven document tasks are answered without any tool being called. The loop gives 0.8750 at the narrow budget and 0.9583 at the course budget; at the course budget the only failure is in the right sequence, wrong answer box and comes from a task whose sequence was completed in full being spoiled by an extra call in between. Steps per correct answer come out at 3.4286 for the baseline and 4.2174 at the course budget, so read alone, efficiency rewards the baseline that calls no tool; on the seventeen tasks that need tools the baseline gives zero correct answers. The run-to-run difference is 0.0000 at the baseline and 0.0417 at the narrow budget.
  2. 02 Testing Each Tool Separately from the Loop Counting how many of eleven fictional defects placed in five tools the tool-level test catches: the eleven-case test suite fails on all four value defects and all three contract defects, but passes clean on all four definition defects. The overall catch rate is 0.6364, and by class 1.0000 and 1.0000 against 0.0000. The mechanism of the unseen class is written down: the test suite calls the tool's function directly, while the loop goes through the field list the tool declares, and a definition defect lives only in that list. The payoff of the layer is independence from the run: on the same task set the task the loop fails shifts from G23 to G15 when the seed changes, while the set of failing cases of the test suite is empty and identical in both runs.
  3. 03 Scenario-Based End-to-End Validation Counting side by side the defect sets caught by two testing layers: the twenty-four-scenario end-to-end flow test catches three of the four definition defects that the tool-level test cannot see at all, and fails on five of the eleven defects in total. The tool-level layer catches seven, the intersection of the two layers is two, and the union is ten of eleven defects, that is 0.9091. The two separate causes of a miss are told apart by the defect activation count: D2, D4, and D11 never activate once during the loop; D5, D6, and D7 activate but the scenario result does not change. The residual defect D11 is a defect whose function is correct but whose definition declares a field it never uses, and no run can see it. On the defect-free baseline the failing scenario in the two runs is G23 and G15.
  4. 04 Monitoring and Logging Measuring what the step trace of an agent loop running in production reads back: in a 240-run stream made of ten runs of twenty-four tasks, fifteen tasks end wrong, and their causes are twelve wrong tool selections and three cases of wandering. The trace that stores only the result gives 0.0000 of these causes, the trace that counts tool names gives 0.4000, and the trace that stores the full observation gives 1.0000; the stored token rises from 2738 to 5954 and 13623, while the token per cause found falls from 992.3 to 908.2. 0.8689 of the sixty-one tool errors occur in runs that end correctly, that is, the tool error counter is a symptom, not a cause.
  5. 05 Cost and Latency Control Reading the spending of an agent loop with two separate denominators: the single-step call without tools is the cheapest at 90.00 cost units per step, but because its success is 0.0000, its cost per correct answer is undefined. When the large and small model classes are run with three step budgets, the small class stays below the 118.30 of the large class at 79.25 per step, while at 488.0 per correct answer it rises above its 453.5. The band where the two rankings diverge is given by the unit price ratio: the break-even ratio is 0.8957 per step and 0.5576 per correct answer, and at every price ratio in between the two measures give opposite answers. Latency is written as a class and read from the longest run; raw time is not measured.
  6. 06 Error Recovery Measuring what a failed tool call does to the loop with three recovery policies: in a production stream of two hundred forty runs, stopping at the first error gives a task success of 0.7500, 727 steps, 53 tool errors, and 81996 tokens; retrying the same tool raises success to 0.9375 and moves the steps to 900, the tool errors to 61, and the tokens to 106352. Dropping the failed tool and moving on to another does the opposite of what is expected: tool errors rise to 64, steps to 907, and success falls to 0.8917. The step efficiency that stopping gains stays within the band of the two runs. The lesson is the close of the AI Agents course and collects the without-scaffolding and with-scaffolding numbers of twenty-two lessons in a single table.

Final exam

AI Agents — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close