Unit 03 of 04
AI in the Development Workflow
Counting from the defensive side what a code assistant gains on a fictional codebase and what it misses: acceptance rate by task class across completion, explanation, and transformation uses, and the point where a bare call collapses without scaffolding; tokens and hit rate as the number of files grows, and the point where irrelevant context lowers the hit rate; how often a multi-step handoff must stop for an error to be caught before it spreads; the defect each of the acceptance check suite, the pattern scan, and the fabricated dependency existence check catches, and the residual none of them catches; the rate at which generated tests catch planted defects and the share of vacuous tests; the number of fields that fall into a secret class in text that enters the context, how many a filter removes, and the share of wrong among accepted suggestions.
Published Updated
Lessons
- 01 Code Assistants Measuring completion, explanation, and transformation uses on a fictional codebase by acceptance rate: the call without context leads only seven of twenty-four tasks to acceptance, and the collapse is in completion (1/8) and transformation (2/8); the only class left standing is explanation (4/8). Open-file context gives 10/24, but the gain is narrow because the run-to-run difference of the same setup is 2/24. The whole codebase raises it to 18/24 at a cost of 1104 tokens rising to 4224 tokens; acceptance per thousand tokens falls from 6.34 to 4.26. In the run-independent expected acceptance column, transformation, even with the whole codebase (0.4675), stays below explanation with no context at all (0.6450). 9 min read
- 02 Providing Context Measuring enlarging the context against selecting the context side by side on the same fictional codebase: sending all the files raises expected acceptance to 0.6921 and spends 4224 tokens and 120 files, while sending only the required files raises it to 0.8233 and spends 2274 tokens and 45 files. Every irrelevant file takes 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file. The token cost of the context provider is independent of its hit rate and is fixed at 2274, but expected acceptance varies between 0.4960 and 0.8233; for the provider to beat sending the whole codebase, it has to find the required files in about three of four tasks. 13 min read
- 03 Agentic Coding Measuring the checkpoint interval when a multi-step job is handed off: when the 480 steps of sixty fictional jobs are run at six different intervals, stopping at every step gives 478/480 correct steps but asks for 479 stops, stopping every eight steps drops to 60 stops and falls to 355/480. Wasted steps per caught error rise from 1.00 to 5.64; missed errors from 1 to 21. The number of calls is almost constant across the six intervals (between 549 and 598), so the cost of stopping is not calls but human attention. Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9), so the most accurate interval and the cheapest interval are not the same. 11 min read
- 04 Reviewing Generated Code Passing twenty-four generated patches on a fictional codebase through three defensive checks: the acceptance check suite fails five of the six behavioral defect classes, the pattern scan eight of eight findings, and the dependency existence check nine of nine fabricated names. Twenty-six of twenty-eight defects are caught, two slip through as format errors, and one of the missed ones enters the codebase without receiving any warning. 4 min read
- 05 Documentation and Test Generation Measuring the discriminating power of a generated test suite: of fourteen generated tests run on the six mutated versions of the fictional codebase, two fail on clean code, nine of the remaining twelve give the same result on all seven versions, and only three distinguish a defect. Detection is 3/6, the ceiling reachable with the whole pool is 6/6, and a single hand-added alignment item makes it 4/6. In the generated documentation ten claims can be checked by machine, six are right, and one describes a function that does not exist. 3 min read
- 06 Limits and Risks Counting the three risks of AI-assisted development: of the 456 context lines in twenty-four request bundles, 168 fall into a secret class; a filter that looks at field names removes all of them but cannot reach the 12 traces left in free text, so its coverage stays at 0.9333; nine of the generated pieces of code carry a dependency name that does not exist; and ten of the nineteen accepted suggestions are defective, so the share of wrong among the accepted is 0.5263, above the baseline share of 0.4583. 14 min read