Unit 02 of 03
Tools and Memory
Counting one by one the surfaces the agent touches and the state it carries: the valid call rate and tool errors across the four levels of a tool definition (name only, name and description, schema, schema and example call); the effect of giving a result raw, truncated, or summarized on success and on accumulated tokens; the number of calls that exceed the declared resource limit, with the share of tasks the limit stops and the share that stopping lowers; completed tasks and stopped calls under full, read-only, and allowlisted permission sets; the share of correct answers and the number of steps carrying the wrong source in a mix of three source classes; the tokens carried per step as the loop grows, the steps that fit in the window, and the information that drops when the window fills; the records stored under the every-step, result-only, and selective write policies and the steps gained in later tasks; the tokens that fall as the compaction ratio rises and the number of tasks that need the discarded field; the stale records that fall and the valid-record loss that rises as the expiry threshold tightens.
Published Updated
Lessons
- 01 Tool Definition Measuring what a tool's name, description, and schema add to the loop: on 24 fictional tasks the definition is given at four levels, and when only the name is given, 113 of 157 calls come back as tool errors, the valid call rate stays at 0.2803, and task success is 0.3333. Adding the description brings the errors down to 56 and raises task success to 0.6667. Adding the schema drops the call count to 74 and the errors to 8, and the valid rate becomes 0.8919, but task success stalls at 0.7917: the schema gives the field names, not which field carries which value, and two swapped indexes return 99948 instead of 52 in a call that passes the schema. The example call zeroes this share, raises task success to 0.9583, and changes the steps, the calls, and the tool errors not at all; its cost is the catalog growing from 97 to 164 tokens, and tokens per correct answer rising from 1126 to 1212. 7 min read
- 02 Executing Tool Calls Measuring in which form the result a tool returns is given back to the model: on 24 fictional tasks the raw observation gives a task success of 0.9583 with 770 observation tokens and 10103 total tokens. Truncating the observation to seven words drops the tokens to 9160, but because it drops the subscriber type, four commercial invoices come out at exactly half and task success falls to 0.7917. Truncating to four words breaks the chain: tool errors jump from 8 to 116, steps from 97 to 156, total tokens to 15945, and task success stays at 0.2917; making it cheaper makes it more expensive. The summary that declares the fields to carry keeps the task success of the raw observation in both runs while dropping the observation tokens to 401 and the tokens per correct answer from 439 to 401. When the subscriber type is removed from the same summary, task success falls to the truncation level, 0.7917: a summary is only as good as its declaration. 12 min read
- 03 Code Execution Tools Measuring, through its resource limits, a tool that runs the program an agent wrote inside a sandbox: three ways of writing a program that computes the same amount spend 15, 172, and 19 steps, hold 2 to 5 cells, and produce 1 to 5 outputs. At the loose limit no call exceeds the declared limit, but three tasks pass the 96-step task budget while the per-call limit stops nothing. The medium limit stops 24 of 28 code calls, and the classes it stops are 20 for steps and 4 for cells; at the tight limit the same 24 calls stop this time on 20 step and 4 output classes. Stopping does not drop a task directly; it drops it by consuming the step budget: on code tasks task success falls from 0.7778 to 0.3333, and in the second run from 1.0000 to 0.5556. The tight limit gives the same task success as the medium limit while cutting wasted steps from 1356 to 552. Running code is entirely on the defensive side. 10 min read
- 04 File System and Terminal Access Counting permission restriction on the agent's file and command surface: twenty-four fictional surface tasks are run under three permission sets, and completed tasks and stopped calls are printed side by side. Under full permission 0.9583 of the tasks are completed and the gate stops no call; the read-only permission stops 71 calls but drops completed tasks to 0.5000; the allowlisted set stops 20 calls and stays at 0.8333, and all of its stops are gathered in a single task kind. Under read-only the correct answer is 0.6250 while completed tasks are 0.5000: in three tasks the answer was produced correctly but the write never happened. A stopped call also consumes a step; steps rise from 78 to 129. Although the totals stay the same in the two runs, two tasks swap places at the task level, so the measurement band is 0.0833 and the 0.1250 difference between full permission and the allowlist is above the band. 11 min read
- 05 Web Search and Browsing Counting access to outside information as a source classification: eighteen fictional sources are classified as primary, secondary, and unverified, and twenty-four outside-information tasks are run in five setups. The loop without outside information gives 0.2083 correct answers; the primary source alone raises it to 0.7500; adding the secondary class to the mix drops it to 0.3750 and pollutes ten tasks; the three classes together raise the polluted tasks to fifteen. On the same pool, ranking the choice by class brings the correct answer back to 0.7500, lowers the polluted tasks to five, and leaves the call count unchanged. After a wrong code enters the state it is carried for an average of 3.40 to 4.00 steps, and in all of the polluted tasks it reaches the record-writing step. The measurement band is 0.0833, so both the 0.3750 drop and the 0.3750 recovery are measured. 13 min read
- 06 Short-Term Memory Counting step by step the conversation state that the agent loop carries: in a single task the tokens carried are 79 at the first step and rise to 255 at the seventh, and most of the growth comes not from the state but from the trace (at the seventh step the state is 63 tokens and the trace load is 192). Twenty-four tasks are run at five window values. With an unlimited window the correct answer is 0.9583, the total is 9316 tokens, and tool errors are 8. At a window of 200 the correct answer falls to 0.8750, at 150 to 0.8333, at 120 to 0.7083; the dropped fields rise from 0 to 85 and tool errors from 8 to 50. The fields dropped most are the identity fields of the task: abone 16, anahtar 16, sutun 15. Narrowing the window does not lower the tokens carried, it raises them: because steps rise from 97 to 112, the total climbs from 9316 to 12026. The measurement band is 0.0833, so the 0.0833 drop at the window of 200 does not count as a finding. 13 min read
- 07 Long-Term Memory Measuring three write policies on the same second set of 48 tasks: making the observation of every step a separate record accumulates 74 records and saves 44 steps, writing a single answer record per task saves 199 steps with 24 records but drops task success from 0.9167 to 0.5833, and the selective policy, which merges the observations in the plan into a single record, saves 118 steps, again with 24 records. What pays off is not the number of records but the unit of the record. 14 min read
- 08 Context Compression Sweeping two compaction policies over the same 24 tasks: when the carried state is paid for as well, the baseline gives 21913 tokens and 0.9583 task success. When selective recall cuts the kept fields to three, tokens fall to 16569 and task success does not change at all, but the number of tasks that need the dropped field rises from 0 to 8. Summarization falls to 0.7917 on the same budget and at the tightest setting raises tokens to 33566, 1.53 times the baseline. This is the point where compaction makes the job worse. 12 min read
- 09 Memory Decay Sweeping the expiry threshold over seven settings: in memory without a threshold 0.4528 of the reads are stale and task success is 0.6389. As the threshold tightens, the stale share falls to 0.0000 and task success rises to 0.9306, but the share of dropped records that are still valid climbs from 0.0000 to 0.9701 and steps rise from 138 to 315. The task success gained by tightening the threshold below eight stays as large as the run-to-run difference itself, that is, it counts as unmeasured. 14 min read