--- title: 'Agentic Coding' source: 'https://academia.sh/en/courses/model-apis/agentic-coding' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows how an error spreads to later steps in a multi-step handoff and where the checkpoint cuts it off.' - 'Compares missed errors, wasted steps, and correctly delivered steps by varying the checkpoint interval.' - 'Calculates the cost of stopping separately as stop count, calls, and tokens.' - 'Interprets why the interval with the highest accuracy and the interval that is cheapest per unit are different.' --- # Agentic Coding Measuring the checkpoint interval when a multi-step job is handed off: when the 480 steps of sixty fictional jobs are run at six different intervals, stopping at every step gives 478/480 correct steps but asks for 479 stops, stopping every eight steps drops to 60 stops and falls to 355/480. Wasted steps per caught error rise from 1.00 to 5.64; missed errors from 1 to 21. The number of calls is almost constant across the six intervals (between 549 and 598), so the cost of stopping is not calls but human attention. Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9), so the most accurate interval and the cheapest interval are not the same. In the previous two lessons every task was **a single call**: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and in the same place as the suggestion. When reading a file, changing a function, updating the callers, and reformatting the report are delegated with a single request, this arrangement breaks: the tool takes steps one after another, each step is built **on top of** the previous one, and the developer no longer looks at every step but **at the place where they stop**. This lesson measures that stopping place and nothing else. How the tool plans, what it remembers, and how it closes its loop are not covered here; those are the subject of the AI Agents course of this curriculum. The only question here is this: **when stopping every how many steps, is the error caught before it propagates, and what does that cost.** - **AD17.** The setup is the same as in the previous two lessons: the **fictional** codebase (five modules, 53 lines), twenty-four fictional tasks, and the assistant **simulator**. No real agentic coding tool is run. - **AD18.** The multi-step job: **60 fictional jobs, 8 steps each**, **480 steps** in total. The context of each step is **selected**, that is, only the files that step requires are sent. - **AD19. Propagation rule:** if a rejected step goes unnoticed, the later steps are built on top of it and are counted as **contaminated**. This is the only structural fact this lesson measures. - **AD20.** A **checkpoint** is placed every `d` steps and at the end of the job; at the stop, the human looks at that window. - **AD21. The detection share falls as the window grows:** `0.95 - 0.07 x (d - 1)`, lower bound `0.40`. Reading an eight-step change in one go is harder than reading a single step. - **AD22.** A caught error is **rerun**: the steps from the faulty step to the stop are called again and assumed to be fixed. These are **wasted steps**. - **AD23.** An error that is not caught **enters the baseline**: the remaining steps of the job are contaminated and later stops cannot see it, because what is looked at is not the accepted baseline but the new work. - **AD24.** The cost has three separate items: **stop count** (the number of times the human looks), **calls**, and **tokens**; each stop also adds a **50-token** status summary. - **AD25.** The delivery measure is the **correct step**: in a job with a missed error, the number of correct steps extends only up to the place where the error is. - **AD26.** Step results are produced with the same seed independently of the interval; the only thing that changes between intervals is **where it stops**. The second run uses seed **20260219**. ## The Trace of a Job First a single job is run with two different intervals. The same eight steps, the same results; only the place of the stops changes. ```python # THE SAME FICTIONAL SETUP -- codebase of five modules (53 lines), 24 fictional tasks, # assistant simulator. No real agentic coding tool is run. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = { 0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py"), } TASKS = [ {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8) ] BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def assistant(task, context, seed=SEED): # SIMULATOR -- FICTIONAL r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context)))) overlap = sum(1 for f in task["required"] if f in context) / len(task["required"]) irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) accepted = r() < score tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0) return {"accepted": accepted, "score": score, "tokens": tokens} JOB_COUNT, STEPS, STATUS_TOKENS = 60, 8, 50 def detection(d): """The human's share of seeing the error falls as the checkpoint window grows.""" return max(0.95 - 0.07 * (d - 1), 0.40) def handoff(j, d, seed=SEED, verbose=False): """Running a single multi-step job with a stop every d steps.""" count = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0, "missed": 0, "wasted": 0, "correct": 0} faulty, escaped = None, False for k in range(STEPS): task = TASKS[(j * STEPS + k) % len(TASKS)] result = assistant(task, task["required"], seed ^ mix("adim|" + str(j) + "|" + str(k))) count["calls"] += 1 count["tokens"] += result["tokens"] if escaped: if verbose: print(f" step {k} {task['name']:<14} contaminated (on top of a missed error)") continue if faulty is None and not result["accepted"]: faulty = k if verbose: label = "accept" if result["accepted"] else "REJECT" if faulty is not None and faulty < k: label = "contaminated" print(f" step {k} {task['name']:<14} {label}") if (k + 1) % d == 0 or k == STEPS - 1: count["stops"] += 1 count["tokens"] += STATUS_TOKENS if faulty is None: if verbose: print(f" STOP step {k}: clean") continue r = rng(seed ^ mix("denetim|" + str(j) + "|" + str(k))) if r() < detection(d): redo = k - faulty + 1 count["caught"] += 1 count["wasted"] += redo count["calls"] += redo count["tokens"] += redo * result["tokens"] if verbose: print(f" STOP step {k}: error at step {faulty} was seen," f" {redo} steps rerun") faulty = None else: count["missed"] += 1 escaped = True if verbose: print(f" STOP step {k}: not seen, the error entered the baseline") count["correct"] = faulty if escaped else STEPS return count print("job 0, step interval 4:") a = handoff(0, 4, verbose=True) print(" ->", {x: a[x] for x in ("correct", "wasted", "stops", "calls", "tokens")}) print("\njob 0, step interval 8:") b = handoff(0, 8, verbose=True) print(" ->", {x: b[x] for x in ("correct", "wasted", "stops", "calls", "tokens")}) ``` ``` job 0, step interval 4: step 0 tamamlama-1 accept step 1 tamamlama-2 accept step 2 tamamlama-3 REJECT step 3 tamamlama-4 contaminated STOP step 3: error at step 2 was seen, 2 steps rerun step 4 tamamlama-5 accept step 5 tamamlama-6 accept step 6 tamamlama-7 REJECT step 7 tamamlama-8 contaminated STOP step 7: error at step 6 was seen, 2 steps rerun -> {'correct': 8, 'wasted': 4, 'stops': 2, 'calls': 12, 'tokens': 1178} job 0, step interval 8: step 0 tamamlama-1 accept step 1 tamamlama-2 accept step 2 tamamlama-3 REJECT step 3 tamamlama-4 contaminated step 4 tamamlama-5 contaminated step 5 tamamlama-6 contaminated step 6 tamamlama-7 contaminated step 7 tamamlama-8 contaminated STOP step 7: error at step 2 was seen, 6 steps rerun -> {'correct': 8, 'wasted': 6, 'stops': 1, 'calls': 14, 'tokens': 1312} ``` The two traces carry the same eight steps and the underlying results are also the same: steps `2` and `6` are rejected. The only thing that changes is the place of the stops, but the trace is not the same. In the four-step interval the first error is caught after spreading two steps, and so is the second; four steps are rerun in total. In the eight-step interval the first error spreads **six steps** and the second error is **never seen** — step `6` is already contaminated at that point, so it is not counted as a separate error. Both deliver eight correct steps in the end, but one spends 12 calls and 1178 tokens, the other 14 calls and 1312 tokens. This single job is a good example but it is not a number: here the check saw the error in both. What happens when it does not see can only be read over enough jobs. ## Sweeping the Interval Sixty jobs, four hundred and eighty steps, six different checkpoint intervals. Step results are independent of the interval; the only thing that changes is the place of the stops. ```python def run_all(d, seed=SEED): total = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0, "missed": 0, "wasted": 0, "correct": 0} for j in range(JOB_COUNT): for key, v in handoff(j, d, seed).items(): total[key] += v return total TOTAL_STEPS = JOB_COUNT * STEPS HEADER = (" every detect stops caught missed waste right step calls tokens tok/right") def show(d, seed): a = run_all(d, seed) print(f"{d:>6}{detection(d):>7.2f}{a['stops']:>7}{a['caught']:>11}{a['missed']:>7}" f"{a['wasted']:>6}{a['correct']:>8}/{TOTAL_STEPS}{a['calls']:>7}{a['tokens']:>10}" f"{a['tokens'] / a['correct']:>11.1f}") print("run A (seed 20260218)") print(HEADER) for d in (1, 2, 3, 4, 6, 8): show(d, SEED) print("\nrun B (seed 20260219)") print(HEADER) for d in (1, 2, 3, 4, 6, 8): show(d, SEED + 1) ``` ``` run A (seed 20260218) every detect stops caught missed waste right step calls tokens tok/right 1 0.95 479 90 1 90 478/480 570 78648 164.5 2 0.88 233 69 8 109 455/480 589 67790 149.0 3 0.81 172 52 11 104 441/480 584 64746 146.8 4 0.74 117 45 11 118 430/480 598 63320 147.3 6 0.60 104 23 23 69 367/480 549 57784 157.4 8 0.46 60 14 21 79 355/480 559 56414 158.9 run B (seed 20260219) every detect stops caught missed waste right step calls tokens tok/right 1 0.95 467 71 4 71 463/480 551 76070 164.3 2 0.88 224 52 11 79 430/480 559 64318 149.6 3 0.81 168 44 12 78 428/480 558 62010 144.9 4 0.74 107 29 14 85 392/480 565 59550 151.9 6 0.60 105 24 17 79 393/480 559 58790 149.6 8 0.46 60 18 16 101 393/480 581 59032 150.2 ``` ## The Measure of Propagation The first half of the question — is the error caught before it propagates — is the quotient of two columns in the table. Wasted steps divided by the number of caught errors give **how many steps the error spread before it was caught**. In the first run this number is, in order, **1.00 / 1.58 / 2.00 / 2.62 / 3.00 / 5.64**; in the second run **1.00 / 1.52 / 1.77 / 2.93 / 3.29 / 5.61**. The two runs say the same thing separately: when stopping at every step the error **does not spread**; when stopping every eight steps it is caught on average after spreading **five and a half steps**. The missed-error column gives the second half. Stopping at every step misses **1** error in the first run and **4** in the second; stopping every eight steps misses **21** and **16**. The reason is written in the table: the detection share falls from 0.95 to 0.46, because reading an eight-step change in one go is hard. One column is misleading here and should not be read: the **number of caught errors**. As the interval grows it falls from 90 to 14, as if the errors had decreased. What decreases is not the errors but **the work in which we can count errors**; since a missed error contaminates the rest of the job, no other error is counted in that job. The quality of delegated work cannot be measured by "how many errors were found". The only thing that can be measured is the **correct steps delivered**: from 478/480 to 355/480, and in the second run from 463/480 to 393/480. The same dependence shows in the stop column. In the run that stops at every step the number of stops is not 480 but **479**; in the second run **467**. The missing stops are the remainder of jobs whose error was missed: because the baseline is already contaminated, there is no clean work left to look at in those stops and they are not counted. The columns of the table are not independent of each other, because the delegated job is not made of independent steps either. ## The Cost of Stopping The cost has three items and the three do not go in the same direction. The **number of calls** stays between 549 and 598 across the six intervals, and between 551 and 581 in the second run — that is, almost constant. Stopping often creates reruns, stopping rarely creates longer reruns; the two cancel each other out. **Stopping is not a matter of calls.** **Tokens** fall steadily: from 78648 to 56414, that is, stopping rarely saves about **twenty-eight percent** of the total tokens. Most of this comes from the 50-token status summary added per stop; the difference between 479 stops and 60 stops alone is 20950 tokens. The real cost is in the third item and it is not paid in tokens: the **stop count**. Stopping at every step calls the human to the desk 479 times, stopping every eight steps 60 times. The difference is **eightfold**, and what is gained in return is 123 correct steps. For every correct step saved, the human has looked about **3.4 times** more. This trade-off is an engineering decision and it is written in the table; which side to choose depends on how reversible a step is. The last column shows a third optimization point. Tokens per correct step are lowest at the **three-step interval** in both runs: **146.8** and **144.9**. This is not the interval that gives the most correct steps (one); nor is it the interval that asks for the fewest stops (eight). Three separate criteria, three separate winners. Choosing a checkpoint interval is therefore not an optimization but a **declaration**: an interval chosen without saying which cost it lowers is not counted as measured. ## Summary - In a multi-step handoff, an error that is not caught spreads to later steps; the checkpoint is where this contamination is cut, and the only thing it measures is **where it stops**. - The number of steps spread before being caught is 1.00 when stopping at every step and 5.64 when stopping every eight steps; 1.00 and 5.61 in the second run. - Missed errors rise from 1 to 21 (from 4 to 16 in the second run), because the detection share falls from 0.95 to 0.46 as the window grows. - The number of caught errors is not a quality measure: it falls as the interval grows, because a missed error makes the rest of the job uncountable. The only thing that can be measured is the correct steps delivered (478/480 against 355/480). - The three items of cost separate: the number of calls is almost constant (between 549 and 598), tokens fall by twenty-eight percent, the stop count falls eightfold. The real price of frequent stopping is human attention. - Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9); the most accurate interval, the cheapest interval, and the interval with the fewest stops are three separate rows. ## Next Step In this lesson the checkpoint was treated as a **place**, and the detection share was squeezed into a single number: the human who stops sees the error with a probability, and if they do not, the error enters the baseline. What was inside that number was never opened. What exactly is looked at at the stop — is the code running, is it giving the right result, is it calling a dependency that does not exist, what was carried in the text that entered the context. The checkpoint was placed, but **what will be checked** was not said. The next lesson fills this gap and turns the detection share from an assumption into a measured number: independent checks are run on the same generated code, the defect each one catches is counted separately, and the **residual** that none of them catches is written by name.