Agentic Coding

In the previous two lessons every task was a single call: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and in the same place as the suggestion. When reading a file, changing a function, updating the callers, and reformatting the report are delegated with a single request, this arrangement breaks: the tool takes steps one after another, each step is built on top of the previous one, and the developer no longer looks at every step but at the place where they stop.

This lesson measures that stopping place and nothing else. How the tool plans, what it remembers, and how it closes its loop are not covered here; those are the subject of the AI Agents course of this curriculum. The only question here is this: when stopping every how many steps, is the error caught before it propagates, and what does that cost.

• AD17. The setup is the same as in the previous two lessons: the fictional codebase (five modules, 53 lines), twenty-four fictional tasks, and the assistant simulator. No real agentic coding tool is run.
• AD18. The multi-step job: 60 fictional jobs, 8 steps each, 480 steps in total. The context of each step is selected, that is, only the files that step requires are sent.
• AD19. Propagation rule: if a rejected step goes unnoticed, the later steps are built on top of it and are counted as contaminated. This is the only structural fact this lesson measures.
• AD20. A checkpoint is placed every d steps and at the end of the job; at the stop, the human looks at that window.
• AD21. The detection share falls as the window grows: 0.95 - 0.07 x (d - 1), lower bound 0.40. Reading an eight-step change in one go is harder than reading a single step.
• AD22. A caught error is rerun: the steps from the faulty step to the stop are called again and assumed to be fixed. These are wasted steps.
• AD23. An error that is not caught enters the baseline: the remaining steps of the job are contaminated and later stops cannot see it, because what is looked at is not the accepted baseline but the new work.
• AD24. The cost has three separate items: stop count (the number of times the human looks), calls, and tokens; each stop also adds a 50-token status summary.
• AD25. The delivery measure is the correct step: in a job with a missed error, the number of correct steps extends only up to the place where the error is.
• AD26. Step results are produced with the same seed independently of the interval; the only thing that changes between intervals is where it stops. The second run uses seed 20260219.

The Trace of a Job

First a single job is run with two different intervals. The same eight steps, the same results; only the place of the stops changes.

# THE SAME FICTIONAL SETUP -- codebase of five modules (53 lines), 24 fictional tasks,
# assistant simulator. No real agentic coding tool is run.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CLASSES = ("tamamlama", "aciklama", "donusturme")
FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py")
REQUIRED = {
    0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"),
    3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"),
    5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"),
    7: ("denetim.py", "bicim.py"),
}
TASKS = [
    {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)}
    for c in CLASSES
    for i in range(8)
]
BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18}


def assistant(task, context, seed=SEED):   # SIMULATOR -- FICTIONAL
    r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context))))
    overlap = sum(1 for f in task["required"] if f in context) / len(task["required"])
    irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES)
    score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"]
    score = min(max(score, 0.0), 1.0)
    accepted = r() < score
    tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0)
    return {"accepted": accepted, "score": score, "tokens": tokens}


JOB_COUNT, STEPS, STATUS_TOKENS = 60, 8, 50


def detection(d):
    """The human's share of seeing the error falls as the checkpoint window grows."""
    return max(0.95 - 0.07 * (d - 1), 0.40)


def handoff(j, d, seed=SEED, verbose=False):
    """Running a single multi-step job with a stop every d steps."""
    count = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0,
             "missed": 0, "wasted": 0, "correct": 0}
    faulty, escaped = None, False
    for k in range(STEPS):
        task = TASKS[(j * STEPS + k) % len(TASKS)]
        result = assistant(task, task["required"], seed ^ mix("adim|" + str(j) + "|" + str(k)))
        count["calls"] += 1
        count["tokens"] += result["tokens"]
        if escaped:
            if verbose:
                print(f"  step {k}  {task['name']:<14} contaminated (on top of a missed error)")
            continue
        if faulty is None and not result["accepted"]:
            faulty = k
        if verbose:
            label = "accept" if result["accepted"] else "REJECT"
            if faulty is not None and faulty < k:
                label = "contaminated"
            print(f"  step {k}  {task['name']:<14} {label}")
        if (k + 1) % d == 0 or k == STEPS - 1:
            count["stops"] += 1
            count["tokens"] += STATUS_TOKENS
            if faulty is None:
                if verbose:
                    print(f"  STOP step {k}: clean")
                continue
            r = rng(seed ^ mix("denetim|" + str(j) + "|" + str(k)))
            if r() < detection(d):
                redo = k - faulty + 1
                count["caught"] += 1
                count["wasted"] += redo
                count["calls"] += redo
                count["tokens"] += redo * result["tokens"]
                if verbose:
                    print(f"  STOP step {k}: error at step {faulty} was seen,"
                          f" {redo} steps rerun")
                faulty = None
            else:
                count["missed"] += 1
                escaped = True
                if verbose:
                    print(f"  STOP step {k}: not seen, the error entered the baseline")
    count["correct"] = faulty if escaped else STEPS
    return count


print("job 0, step interval 4:")
a = handoff(0, 4, verbose=True)
print("  ->", {x: a[x] for x in ("correct", "wasted", "stops", "calls", "tokens")})
print("\njob 0, step interval 8:")
b = handoff(0, 8, verbose=True)
print("  ->", {x: b[x] for x in ("correct", "wasted", "stops", "calls", "tokens")})

job 0, step interval 4:
  step 0  tamamlama-1    accept
  step 1  tamamlama-2    accept
  step 2  tamamlama-3    REJECT
  step 3  tamamlama-4    contaminated
  STOP step 3: error at step 2 was seen, 2 steps rerun
  step 4  tamamlama-5    accept
  step 5  tamamlama-6    accept
  step 6  tamamlama-7    REJECT
  step 7  tamamlama-8    contaminated
  STOP step 7: error at step 6 was seen, 2 steps rerun
  -> {'correct': 8, 'wasted': 4, 'stops': 2, 'calls': 12, 'tokens': 1178}

job 0, step interval 8:
  step 0  tamamlama-1    accept
  step 1  tamamlama-2    accept
  step 2  tamamlama-3    REJECT
  step 3  tamamlama-4    contaminated
  step 4  tamamlama-5    contaminated
  step 5  tamamlama-6    contaminated
  step 6  tamamlama-7    contaminated
  step 7  tamamlama-8    contaminated
  STOP step 7: error at step 2 was seen, 6 steps rerun
  -> {'correct': 8, 'wasted': 6, 'stops': 1, 'calls': 14, 'tokens': 1312}

The two traces carry the same eight steps and the underlying results are also the same: steps 2 and 6 are rejected. The only thing that changes is the place of the stops, but the trace is not the same. In the four-step interval the first error is caught after spreading two steps, and so is the second; four steps are rerun in total. In the eight-step interval the first error spreads six steps and the second error is never seen — step 6 is already contaminated at that point, so it is not counted as a separate error. Both deliver eight correct steps in the end, but one spends 12 calls and 1178 tokens, the other 14 calls and 1312 tokens.

This single job is a good example but it is not a number: here the check saw the error in both. What happens when it does not see can only be read over enough jobs.

Sweeping the Interval

Sixty jobs, four hundred and eighty steps, six different checkpoint intervals. Step results are independent of the interval; the only thing that changes is the place of the stops.

def run_all(d, seed=SEED):
    total = {"calls": 0, "tokens": 0, "stops": 0, "caught": 0,
             "missed": 0, "wasted": 0, "correct": 0}
    for j in range(JOB_COUNT):
        for key, v in handoff(j, d, seed).items():
            total[key] += v
    return total


TOTAL_STEPS = JOB_COUNT * STEPS
HEADER = (" every detect  stops     caught missed waste  right step  calls    tokens  tok/right")


def show(d, seed):
    a = run_all(d, seed)
    print(f"{d:>6}{detection(d):>7.2f}{a['stops']:>7}{a['caught']:>11}{a['missed']:>7}"
          f"{a['wasted']:>6}{a['correct']:>8}/{TOTAL_STEPS}{a['calls']:>7}{a['tokens']:>10}"
          f"{a['tokens'] / a['correct']:>11.1f}")


print("run A (seed 20260218)")
print(HEADER)
for d in (1, 2, 3, 4, 6, 8):
    show(d, SEED)
print("\nrun B (seed 20260219)")
print(HEADER)
for d in (1, 2, 3, 4, 6, 8):
    show(d, SEED + 1)

run A (seed 20260218)
 every detect  stops     caught missed waste  right step  calls    tokens  tok/right
     1   0.95    479         90      1    90     478/480    570     78648      164.5
     2   0.88    233         69      8   109     455/480    589     67790      149.0
     3   0.81    172         52     11   104     441/480    584     64746      146.8
     4   0.74    117         45     11   118     430/480    598     63320      147.3
     6   0.60    104         23     23    69     367/480    549     57784      157.4
     8   0.46     60         14     21    79     355/480    559     56414      158.9

run B (seed 20260219)
 every detect  stops     caught missed waste  right step  calls    tokens  tok/right
     1   0.95    467         71      4    71     463/480    551     76070      164.3
     2   0.88    224         52     11    79     430/480    559     64318      149.6
     3   0.81    168         44     12    78     428/480    558     62010      144.9
     4   0.74    107         29     14    85     392/480    565     59550      151.9
     6   0.60    105         24     17    79     393/480    559     58790      149.6
     8   0.46     60         18     16   101     393/480    581     59032      150.2

The Measure of Propagation

The first half of the question — is the error caught before it propagates — is the quotient of two columns in the table. Wasted steps divided by the number of caught errors give how many steps the error spread before it was caught. In the first run this number is, in order, 1.00 / 1.58 / 2.00 / 2.62 / 3.00 / 5.64; in the second run 1.00 / 1.52 / 1.77 / 2.93 / 3.29 / 5.61. The two runs say the same thing separately: when stopping at every step the error does not spread; when stopping every eight steps it is caught on average after spreading five and a half steps.

The missed-error column gives the second half. Stopping at every step misses 1 error in the first run and 4 in the second; stopping every eight steps misses 21 and 16. The reason is written in the table: the detection share falls from 0.95 to 0.46, because reading an eight-step change in one go is hard.

One column is misleading here and should not be read: the number of caught errors. As the interval grows it falls from 90 to 14, as if the errors had decreased. What decreases is not the errors but the work in which we can count errors; since a missed error contaminates the rest of the job, no other error is counted in that job. The quality of delegated work cannot be measured by “how many errors were found”. The only thing that can be measured is the correct steps delivered: from 478/480 to 355/480, and in the second run from 463/480 to 393/480.

The same dependence shows in the stop column. In the run that stops at every step the number of stops is not 480 but 479; in the second run 467. The missing stops are the remainder of jobs whose error was missed: because the baseline is already contaminated, there is no clean work left to look at in those stops and they are not counted. The columns of the table are not independent of each other, because the delegated job is not made of independent steps either.

The Cost of Stopping

The cost has three items and the three do not go in the same direction. The number of calls stays between 549 and 598 across the six intervals, and between 551 and 581 in the second run — that is, almost constant. Stopping often creates reruns, stopping rarely creates longer reruns; the two cancel each other out. Stopping is not a matter of calls.

Tokens fall steadily: from 78648 to 56414, that is, stopping rarely saves about twenty-eight percent of the total tokens. Most of this comes from the 50-token status summary added per stop; the difference between 479 stops and 60 stops alone is 20950 tokens.

The real cost is in the third item and it is not paid in tokens: the stop count. Stopping at every step calls the human to the desk 479 times, stopping every eight steps 60 times. The difference is eightfold, and what is gained in return is 123 correct steps. For every correct step saved, the human has looked about 3.4 times more. This trade-off is an engineering decision and it is written in the table; which side to choose depends on how reversible a step is.

The last column shows a third optimization point. Tokens per correct step are lowest at the three-step interval in both runs: 146.8 and 144.9. This is not the interval that gives the most correct steps (one); nor is it the interval that asks for the fewest stops (eight). Three separate criteria, three separate winners. Choosing a checkpoint interval is therefore not an optimization but a declaration: an interval chosen without saying which cost it lowers is not counted as measured.

Summary

• In a multi-step handoff, an error that is not caught spreads to later steps; the checkpoint is where this contamination is cut, and the only thing it measures is where it stops.
• The number of steps spread before being caught is 1.00 when stopping at every step and 5.64 when stopping every eight steps; 1.00 and 5.61 in the second run.
• Missed errors rise from 1 to 21 (from 4 to 16 in the second run), because the detection share falls from 0.95 to 0.46 as the window grows.
• The number of caught errors is not a quality measure: it falls as the interval grows, because a missed error makes the rest of the job uncountable. The only thing that can be measured is the correct steps delivered (478/480 against 355/480).
• The three items of cost separate: the number of calls is almost constant (between 549 and 598), tokens fall by twenty-eight percent, the stop count falls eightfold. The real price of frequent stopping is human attention.
• Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9); the most accurate interval, the cheapest interval, and the interval with the fewest stops are three separate rows.

Next Step

In this lesson the checkpoint was treated as a place, and the detection share was squeezed into a single number: the human who stops sees the error with a probability, and if they do not, the error enters the baseline. What was inside that number was never opened. What exactly is looked at at the stop — is the code running, is it giving the right result, is it calling a dependency that does not exist, what was carried in the text that entered the context. The checkpoint was placed, but what will be checked was not said. The next lesson fills this gap and turns the detection share from an assumption into a measured number: independent checks are run on the same generated code, the defect each one catches is counted separately, and the residual that none of them catches is written by name.