Web Search and Browsing

The previous lesson tied the surface the agent touches to a permission gate: which path would be read and which command would run was a set that we declared. Outside information removes that ground. When the agent finds a record with a search tool and reads its content with a browsing tool, the text that arrives is not ours; it fits neither an allowlist nor a schema.

Only one question can be asked of information that comes from outside: who kept this record. The lesson turns that question into a source classification and counts two things: how the mix moves the share of correct answers, and how many steps a wrong record is carried after it enters the state.

• TM37. The code is a simulator; no real search or browsing tool is called. The sources, the claims, and the questions are fictional, seed 20260218.
• TM38. Sources are named only by their class. No site, domain name, institution, or country name is written; a source is nothing more than a fictional record number and a class.
• TM39. There are three classes: primary (the operator’s own record), secondary (a derived compilation), unverified (open contribution). The declared source reliability shares are 0.95, 0.70, and 0.30 and are fictional; what is measured is not the value of the share but the result of the mix.
• TM40. Each of the eighteen fictional sources covers a single topic and on that topic either carries the correct record or does not; a source that does not carries the code of another topic.
• TM41. The twenty-four tasks are spread over six topics; two topics exist in the internal document store, and four are only outside.
• TM42. The task sequence is three steps: search, fetch, write to the record. The wrong code does not stay in the answer, it can also reach the step with a side effect.
• TM43. The baseline is the loop without outside information — the agent looks only at the internal document store. The cost is written as tool calls, steps, polluted tasks, and the steps over which the wrong code is carried.

The Shared Loop

The environment, the five basic tools, and the agent loop are the same throughout the course.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL environment: records of the measurement network
REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


# ---- FICTIONAL tasks: each one is solved by a known tool sequence
TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


# ---- FICTIONAL agent: at each step it picks a tool or finishes
def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}


print("shared environment:", len(SUBSCRIBERS), "subscribers ,", len(DOCUMENTS), "documents ,",
      len(TOOLS), "tools ,", len(TASKS), "tasks")

shared environment: 60 subscribers , 5 documents , 5 tools , 24 tasks

Source Class, Mix, and Choice

Two tools are added to the surface: web_search returns the records that match the topic, and fetch_page reads the code the chosen record carries. The only design decision the search tool makes is this: which classes are in the mix and which one is chosen from the candidate list. The choice does not verify a record, it looks only at its class.

# ---- FICTIONAL outside information surface: sources are named by class, there is no place name
CODES = ("SY01", "TR01", "BR01", "BR02", "BR03", "BR04")
QUESTION = {"SY01": "sayac donusunde endeks farki nasil hesaplanir",
            "TR01": "tarife basamaklari hangi metrekup degerlerinde degisir",
            "BR01": "bolge dagitim katsayisi kac",
            "BR02": "sayac degisim araligi kac yil",
            "BR03": "itiraz suresi kac gun",
            "BR04": "gecikme orani nasil hesaplanir"}
TOPIC_HINT = {"endek": "SY01", "basam": "TR01", "katsa": "BR01",
              "degis": "BR02", "itira": "BR03", "gecik": "BR04"}
CLASSES = ("primary", "secondary", "unverified")
RELIABILITY = {"primary": 0.95, "secondary": 0.70, "unverified": 0.30}
SOURCES, FETCHED = {}, []
for i in range(18):
    r = rng(SEED + 2593 * i)
    s = CLASSES[pick(r(), [0.28, 0.34, 0.38])]
    SOURCES[f"K{i + 1:02d}"] = {"class": s, "coverage": CODES[int(r() * 6)],
                                "correct": r() < RELIABILITY[s]}
MIX, RANKED = [set(CLASSES)], [False]


def resolve_topic(a):
    return next((TOPIC_HINT[t] for t in tokenize(a) if t in TOPIC_HINT), None)


def t_web_search(query="", **_):
    code = resolve_topic(query)
    candidates = [k for k, v in SOURCES.items()
                  if v["coverage"] == code and v["class"] in MIX[0]]
    if not candidates:
        return {"error": "no source"}
    if RANKED[0]:                       # class-ranked choice: primary first
        candidates.sort(key=lambda k: (CLASSES.index(SOURCES[k]["class"]), k))
    return {"source": candidates[0], "found": len(candidates)}


def t_fetch_page(source=None, **_):
    v = SOURCES.get(source)
    FETCHED.append(None if v is None else v["correct"])    # stays aligned with the trace
    if v is None:
        return {"error": "no source"}
    return {"result": v["coverage"] if v["correct"]
            else CODES[(CODES.index(v["coverage"]) + 3) % 6], "class": v["class"]}


TOOLS.update({"web_search": {"function": t_web_search, "fields": ("query",),
                             "side_effect": False},
              "fetch_page": {"function": t_fetch_page, "fields": ("source",),
                             "side_effect": False}})
TOOL_NAMES = list(TOOLS)

EXTERNAL = []
for i in range(24):
    r = rng(SEED + 4643 * i)
    code = CODES[pick(r(), [0.16, 0.16, 0.18, 0.16, 0.16, 0.18])]
    EXTERNAL.append({"name": f"W{i + 1:02d}", "code": code,
                     "subscriber": f"A{1 + int(r() * 60):03d}",
                     "request": QUESTION[code],
                     "sequence": ("web_search", "fetch_page", "write_record")})


def run(classes, ranked=False, internal=False, seed=SEED):
    """internal=True: no outside information, the agent looks only at the internal document store."""
    MIX[0], RANKED[0] = set(classes), ranked
    CALLS[0] = 0
    correct = steps = polluted = carried = record = 0
    for g in EXTERNAL:
        FETCHED.clear()
        s = agent(dict(g, sequence=("search_docs", "write_record")) if internal else g, seed=seed)
        d = s["answer"] == g["code"]
        correct, steps = correct + d, steps + s["steps"]
        place = [i for i, (a, _) in enumerate(s["trace"]) if a == "fetch_page"]
        wrong = [place[m] for m, ok in enumerate(FETCHED) if ok is False]
        if wrong and not d:
            polluted, carried = polluted + 1, carried + s["steps"] - wrong[-1]
            record += any(a == "write_record" and not h for a, h in s["trace"])
    return correct, polluted, carried / polluted if polluted else 0.0, record, CALLS[0], steps


print("source class distribution:",
      {s: sum(v["class"] == s for v in SOURCES.values()) for s in CLASSES})
print("sources of each class carrying correct information:",
      {s: sum(v["class"] == s and v["correct"] for v in SOURCES.values()) for s in CLASSES})
print("task topic:", {k: sum(g["code"] == k for g in EXTERNAL) for k in CODES})
print("\nsetup                  correct A correct B  polluted tasks  carried steps"
      "  polluted records  tool calls  steps")
for name, cl, rk, inn in (("no outside information", (), False, True),
                          ("primary only", ("primary",), False, False),
                          ("primary + secondary", CLASSES[:2], False, False),
                          ("three classes", CLASSES, False, False),
                          ("three classes , ranked", CLASSES, True, False)):
    a = run(cl, rk, inn)
    b = run(cl, rk, inn, SEED + 1)
    print(f"{name:<22} {a[0] / 24:>9.4f} {b[0] / 24:>9.4f}  {a[1]:>14}"
          f"  {a[2]:>13.2f}  {a[3]:>16}  {a[4]:>10}  {a[5]:>5}")

source class distribution: {'primary': 6, 'secondary': 5, 'unverified': 7}
sources of each class carrying correct information: {'primary': 6, 'secondary': 3, 'unverified': 1}
task topic: {'SY01': 3, 'TR01': 2, 'BR01': 5, 'BR02': 5, 'BR03': 4, 'BR04': 5}

setup                  correct A correct B  polluted tasks  carried steps  polluted records  tool calls  steps
no outside information    0.2083    0.2083               0           0.00                 0          91    110
primary only              0.7500    0.8333               0           0.00                 0         121    139
primary + secondary       0.3750    0.4167              10           3.40                10         121    139
three classes             0.3750    0.3750              15           3.60                15         103    126
three classes , ranked    0.7500    0.7917               5           4.00                 5         103    126

The Number the Mix Changes

The baseline row is the loop without outside information: because the internal document store covers two of the six topics, the correct answer stays at 0.2083. Opening the primary sources raises it to 0.7500 with zero polluted tasks: all six primary sources carry the correct record. The cost is a rise from 91 to 121 calls and from 110 to 139 steps; because the primary class covers four of the six topics, on the remaining two the search returns no source and the agent retries the step.

The third row is the example of the course’s second claim. Adding the secondary class to the mix drops the correct answer from 0.7500 to 0.3750. The mechanism is in the second output line: only three of the five secondary sources carry the correct record, and because the choice does not look at the class, a secondary record can come ahead of a primary one. Adding sources here added noise, not information. Adding the unverified class too leaves the correct answer at 0.3750 but raises the polluted tasks from 10 to 15; in return the calls fall from 121 to 103, because once coverage is complete there is no no source error left. Cheaper and more polluted.

The last row shows where the fix is. The pool is the same, the mix is the same, and the calls stay at 103 and the steps at 126; the only thing that changed is that the candidate list is ranked by class. The correct answer returns to 0.7500, and the polluted tasks fall from 15 to 5. We had reached the same hit rate with the primary source alone, but there it took 121 calls and 139 steps. Ranking the classes, not discarding the unreliable class, is both more correct and cheaper. The five polluted tasks that remain also write down the limit of ranking: the two topics with no primary source at all correspond, in the task distribution, to exactly five tasks.

The two run columns tell which difference is real: the largest run-to-run difference is between 0.8333 and 0.7500 on the primary-only row, and the band is 0.0833. Both the 0.3750 drop and the 0.3750 recovery are far above the band and are measured. The life of the pollution is in the polluted task and carried step columns: after the wrong code enters the state it is carried for an average of 3.40, 3.60, and 4.00 steps, and all of the polluted tasks also appear in the polluted record column. The wrong source does not stop at the answer, it becomes permanent through write_record.

Summary

• In outside information, what can be checked is not the correctness of the content but the class of the record; sources are counted as primary, secondary, and unverified, and none of them is named.
• The loop without outside information gives 0.2083 correct answers; the primary source alone raises it to 0.7500, at the cost of calls rising from 91 to 121.
• Adding the secondary class to the mix drops the correct answer to 0.3750; adding sources can add noise instead of information, and this is an example of the second claim of this course.
• The three classes together raise the polluted tasks to 15 but lower the calls to 103; on the same pool, ranking the choice by class brings the correct answer to 0.7500, lowers the polluted tasks to 5, and does not raise the calls.
• A wrong code is carried for an average of 3.40 to 4.00 steps and reaches the record-writing step in all of the polluted tasks; because the measurement band is 0.0833, both 0.3750 swings are measured.

Next Step

In this lesson the number of steps over which a wrong code is carried was counted, but the thing that carries it was not measured. The state of the agent grows at every step: observations accumulate, the trace lengthens, and it is sent again with every call that follows. The context window was defined in the Introduction to AI Engineering course, and resending the conversation state with every request was measured in the Building Applications with Model APIs course; neither is repeated. The next lesson asks what is specific to the loop: how many tokens are carried per step, how many steps fit in the window, and which field drops when the window fills.