Impact on Product Development

The previous lesson showed the seed row with a single number: the overall hit rate moved from 0.7778 to 0.8194. That row is the hardest design question of a product, and the overall number hides it. The same user, on the same screen, sending the same text twice can get two different responses. Even the overall hit rate staying the same would hide it, because a correct response won on one record may have been lost on another.

This lesson counts uncertainty at the record level and separates two questions: in how many records the output changed and what part of that change is visible to the user. Then the three design decisions that narrow uncertainty are compared on the same measure.

• FR16. The fiction, the simulator, and the test set carry over unchanged from the previous lesson: 72 records, five topics, seed 20260218. The second run is taken with seed 20260219.
• FR17. What is compared between the two runs is the output the product shows. On the plain prompt this is a sentence, on the instructed prompt a single label; the two are counted separately.
• FR18. The fictional document pool is ten records: tariff descriptions, meter maintenance procedures, and fault codes. The document codes are also fictional.
• FR19. Retrieval selects the document that overlaps the most with the ticket’s tokens. It is a step, not a model call; it is counted in a separate column in the cost table.
• FR20. The hit rate of offering options is read on a different criterion: it is enough for the correct label to be among the two candidates. It stands in the same column as the other three rows but does not measure the same thing.
• FR21. The visible change is good when it is low; the hit rate is good when it is high. The two columns point in opposite directions and are read together in the table.
• FR22. Calls that hit the rate limit land on the same places in both runs because the counter is reset at the start of every measurement; the change count therefore comes only from the model’s uncertainty.

Two Runs of the Same Input

The first block sets up the simulator and calls each ticket with two seeds, separating four numbers: did the text change, did the decision change, is it correct in both runs, is it correct in only one.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets are also FICTIONAL.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):
    tok = []
    for s in text.lower().replace(",", " , ").split():
        while len(s) > 5:
            tok.append(s[:5])
            s = s[5:]
        tok.append(s)
    return tok


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["invoice", "fault", "meter", "tariff", "outage"]
TEMPLATES = {
    "invoice": ["the invoice amount this month is very high compared to last month",
                "i paid my invoice but the system shows a debt",
                "where can i see the invoice details"],
    "fault": ["the water is not flowing at home and there is a fault i think",
              "the connection is broken and it is not working at all",
              "i opened a fault ticket but the team never came"],
    "meter": ["the meter reading was entered wrong and the index does not match",
              "i want an appointment for the meter change",
              "the index on the meter and the invoice are not the same"],
    "tariff": ["i want to learn which tariff tier i am on",
               "what do i need to do for a tariff change",
               "does the unit price change by subscriber type"],
    "outage": ["there is an outage in the neighborhood when will it be back",
               "where is the planned outage announcement tracked",
               "we have been having outages since the evening"],
}
EXTRAS = ["the same issue happened last month too", "i am not satisfied with this",
          "i urgently need information", "i do not have my record number"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"]
TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice",
          "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault",
          "meter": "meter", "index": "meter", "readi": "meter",
          "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff",
          "outag": "outage", "annou": "outage", "neigh": "outage",
          "debt": "payment", "paid": "payment", "price": "payment",
          "satis": "complaint", "issue": "complaint", "urgen": "complaint",
          "infor": "information", "learn": "information", "where": "information"}
TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice",
         "team": "outage", "track": "tariff", "enter": "invoice"}
FREE_FORM = ["your request seems to relate to {k}",
             "this falls under {k} , forwarded to the relevant team",
             "your record has been received , we will review the matter and get back to you"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((TRACES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:
        out = out[:max_output]
        y = " ".join(out)
    return {"status": "ok", "input": len(tok), "output": len(out), "response": y}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


TEST = TICKETS[48:]
PLAIN = "Answer this request: "
INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , "
               "outage . Write only the label as the response . Ticket: ")


def two_runs(prefix):
    COUNTER[0] = 0
    text_changed = decision_changed = both = one = neither = 0
    for x in TEST:
        a = call(prefix + x["text"], seed=SEED)
        b = call(prefix + x["text"], seed=SEED + 1)
        ea, eb = read_label(a["response"]), read_label(b["response"])
        text_changed += a["response"] != b["response"]
        decision_changed += ea != eb
        d = (ea == x["topic"]) + (eb == x["topic"])
        both += d == 2
        one += d == 1
        neither += d == 0
    n = len(TEST)
    return text_changed / n, decision_changed / n, both / n, one / n, neither / n


print(f"{'prompt':<10} {'text changed':>14} {'decision changed':>16} {'both correct':>12} "
      f"{'only one':>12} {'both wrong':>13}")
for name, o in (("plain", PLAIN), ("instructed", INSTRUCTION)):
    m, k, i2, b1, h = two_runs(o)
    print(f"{name:<10} {m:>14.4f} {k:>16.4f} {i2:>12.4f} {b1:>12.4f} {h:>13.4f}")

prompt       text changed decision changed both correct     only one    both wrong
plain              0.8750           0.5972       0.2222       0.5556        0.2222
instructed         0.3472           0.3472       0.6389       0.3056        0.0556

In the first row, the difference between two columns is the main finding of the lesson. On the plain prompt, the response text changes in 0.8750 of the records, but the decision underneath changes in only 0.5972. The 0.2778 in between is the records where the decision stayed the same although the sentence was composed differently. If what the user sees is the sentence, the product looks half again as indecisive; if what the user sees is the decision, the real indecision is what gets read. The uncertainty of a product is not the model’s uncertainty but the uncertainty of what it puts on the screen.

The fourth column is the most expensive number from the product’s point of view. On the plain prompt, in 0.5556 of the records the correct response comes out in only one run. In these records, whether the user gets a correct response does not depend on anything the user can see or change. On the instructed prompt this share falls to 0.3056 and the “wrong in both runs” share drops from 0.2222 to 0.0556: the instruction does not merely increase the number of correct responses, it also reduces the persistent errors.

These two rows have three direct consequences for product design. First, sending the same ticket twice is not a bug but expected behavior; storing the response and showing the stored response on the second request brings the visible indecision down to zero without changing the hit rate at all. Second, the “correct in only one” share should be written to the support record: whether a wrong response is persistent or tied to the run can be told apart only if both runs are stored. Third, leaving the text shown to the user free is not free of charge; two different sentences carrying the same decision appear to the user as two different responses and increase the number of complaints independently of the decision itself.

Three Design Decisions That Narrow Uncertainty

The three decisions are compared on the same measure: offering options, imposing a format, grounding. For grounding, a fictional document pool is set up and the document that overlaps the ticket the most is added to the prompt.

DOCUMENTS = [("TR-11", "the tariff tier is determined by subscriber type and unit price"),
             ("TR-12", "tariff change requests are accepted at the start of each period"),
             ("FT-21", "the invoice amount is calculated from the period reading , "
                       "the detail is on screen"),
             ("FT-22", "after payment the debt entry is closed the same day"),
             ("SY-31", "meter index reading is taken each period and goes into the log"),
             ("SY-32", "meter change is done by appointment , the old index stays in the log"),
             ("AR-41", "when a fault ticket is opened the field crew is dispatched"),
             ("AR-42", "for a broken connection the line is checked first"),
             ("KS-51", "the planned outage announcement is published first"),
             ("KS-52", "outage follow-up per neighborhood is done on a map")]


def retrieve(text):                     # RETRIEVAL step: the document with the most overlapping pieces
    b = set(tokenize(text))
    return max(DOCUMENTS, key=lambda d: (len(b & set(tokenize(d[1]))), d[0]))


def plan(design, x, seed):              # prompts and the number of retrieval steps
    if design == "free":
        return [(PLAIN + x["text"], seed)], 0
    if design == "options":
        return [(INSTRUCTION + x["text"], seed + j) for j in (0, 2)], 0
    if design == "format":
        return [(INSTRUCTION + x["text"], seed)], 0
    k, m = retrieve(x["text"])
    return [(INSTRUCTION + "Document " + k + ": " + m + " . Ticket: " + x["text"], seed)], 1


def run(design, seed):
    COUNTER[0] = 0
    shown, correct, tokens, steps = [], 0, 0, 0
    for x in TEST:
        prompts, a = plan(design, x, seed)
        steps, e, last = steps + a, [], ""
        for p, sd in prompts:
            s = call(p, seed=sd)
            tokens += s["input"] + s["output"]
            e.append(read_label(s["response"]))
            last = s["response"]
        labels = sorted(set(e) - {None})
        shown.append(" ".join(labels) if design == "options" else last)
        correct += (x["topic"] in labels) if design == "options" else (e[0] == x["topic"])
    return shown, correct / len(TEST), COUNTER[0], tokens, steps


print(f"{'design decision':<18} {'change':>8} {'hit rate':>8} {'calls':>6} "
      f"{'tokens':>9} {'retrieval':>9}")
for name, d in (("no scaffolding", "free"), ("impose format", "format"),
                ("offer options", "options"), ("grounding", "grounding")):
    g1, i1, c1, b1, a1 = run(d, SEED)
    g2, i2, c2, b2, a2 = run(d, SEED + 1)
    dg = sum(u != v for u, v in zip(g1, g2)) / len(g1)
    print(f"{name:<18} {dg:>8.4f} {(i1 + i2) / 2:>8.4f} {c1 + c2:>6d} {b1 + b2:>9d} "
          f"{a1 + a2:>9d}")

design decision      change hit rate  calls    tokens retrieval
no scaffolding       0.8611   0.5139    144      4921         0
impose format        0.3056   0.7986    144      7514         0
offer options        0.4306   0.9375    288     15013         0
grounding            0.0972   0.8264    144     10835       144

Imposing a format is the cheapest decision: it lowers the visible change from 0.8611 to 0.3056, raises the hit rate from 0.5139 to 0.7986, and does not touch the call count at all. Its cost is 7514 tokens against 4921 for no scaffolding, and all of the added cost comes from the instruction written into the prompt.

Grounding is the strongest decision: the visible change drops to 0.0972, meaning the two runs diverge in fewer than one record in ten. The hit rate is 0.8264. The reason is that once the document enters the prompt, the evidence for the correct topic outweighs the rest, that is, less of the decision is left to sampling. Its cost has two items: 10835 tokens against 7514 for imposing a format, and 144 retrieval steps. Retrieval is not a model call but it is a source of latency and carries its own error — when the wrong document arrives, the model gives a consistently wrong response.

Offering options is the row in this table that pays off the second claim of the curriculum. The highest number in the hit rate column belongs to it: 0.9375. But that number is a different criterion — it is enough for the correct label to be among the two candidates, and the choosing is left to the user. On its own criterion, that is the visible change, offering options sets back imposing a format: from 0.3056 to 0.4306. When one of the two candidates changes, the box on the screen changes too, so the two runs diverge in more records. In return the call count doubles and the tokens double. The added scaffolding made the number worse and its cost definitely rose.

Summary

• On the plain prompt, the response text changes in 0.8750 of the records, while the decision underneath changes in 0.5972. The difference is that, depending on what the product puts on the screen, the indecision appears half again as large.
• In 0.5556 of the records the correct response comes out in only one run. This share is a loss the user can neither see nor influence; on the instructed prompt it falls to 0.3056, and the persistent wrong share falls from 0.2222 to 0.0556.
• Imposing a format moves the visible change from 0.8611 to 0.3056 and the hit rate from 0.5139 to 0.7986 without changing the call count; its cost is 7514 tokens against 4921.
• Grounding lowers the change to 0.0972 and makes the hit rate 0.8264; its cost is 10835 tokens against 7514, and 144 retrieval steps. When the wrong document arrives, the wrong response becomes stable.
• Offering options raises the visible change from 0.3056 to 0.4306 and doubles the calls; the 0.9375 in the hit rate column is a different criterion and hands the choosing to the user.
• The latency class also separates: the no-scaffolding and format-imposing paths are a single call, grounding is a chain of retrieval and call, offering options is two parallel calls. The four rows are read with a single rule: a design decision can be compared only when it is written on its own criterion and with its cost.

Next Step

Over three lessons the model was treated as a given component, measured from its surface, its settings were counted, and its uncertainty was translated into product language. The words used throughout these three lessons are also a design object. Expressions such as “capability”, “understands”, “learns”, and “general intelligence” are things too different to be written into the same table: some of them can be tied to a test and give a number, and some can never be falsified under any condition. The next lesson separates a set of sentences by this criterion and counts how many can actually be tied to a test.