Inference

The previous lesson separated where the model learns what from, but it produced a single prediction everywhere: the most likely token, in a single step. A real response is not a single token. The model produces a token, appends it to the end of its own input, produces the next one, and keeps going until the stop condition arrives. What we call inference is this loop; both its cost and its uncertainty are born here.

Two questions are measured. The first is cost: how much work is done as a response gets longer. The second is the selection rule: at each step, is the largest of the distribution taken, or is the choice sampled from the distribution — and what does this decision give in two runs on the same prompt. This lesson directly counts the third claim of the curriculum: the model is an uncertain component and the amount of uncertainty is a setting.

• LM26. The tickets and the response samples are fictional: they are generated inside the lesson, seed 20260218. The sample set is 600 responses, the test set 200 tickets.
• LM27. The model is a simulator: at every step it builds a distribution for the condition and the last two tokens, chooses a token and appends its choice to the history.
• LM28. The stop condition is the period. The maximum output length is a separate limit and when it is exceeded the response is cut midway.
• LM29. 0.15 of the response samples are a general pattern unrelated to the domain. For this reason the model does not have a single correct path under every condition.
• LM30. Hit rate is counted only for completed responses: the response must end with a period and carry the correct object and the correct action. A response cut midway counts as wrong.
• LM31. Cost is written in two units: the number of tokens produced and the number of comparisons. Each step costs as many comparisons as the input length plus the tokens produced up to that point.
• LM32. The input length is 30 tokens in all measurements; the only thing that changes is the output.
• LM33. Distribution temperature raises the probabilities to a power: zero is greedy decoding, and as it grows the distribution flattens.
• LM34. The measure of uncertainty is two runs of the same prompt: the second run is taken by changing the seed and the share of responses that come out different in the two runs is printed.

Producing a Response

The simulator does not write a response at once. It starts with the start marker, builds a distribution at each step, chooses a token and appends its choice to the end of the history.

# SIMULATOR — the model is a simulator written in the lesson with the standard
# library; no real endpoint is called. Tickets and response samples are FICTIONAL, seed 20260218.
SEED, M32, SAMPLES, TEST = 20260218, 0xFFFFFFFF, 600, 200


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


OBJECTS = ["invoice", "tariff", "meter", "valve"]
STATUSES = ["high", "missing", "broken"]
ACTIONS = {"high": "objection record opened", "missing": "information sent",
           "broken": "field team dispatched"}
TEMPLATES = ["{n} for {e} .", "{n} record examined and {e} .",
             "{n} condition for {e} ."]
GENERIC = ["issue examined and result sent .", "request received and issue opened ."]


def make_samples(seed, count):
    v = []
    for i in range(count):
        r = rng(seed + i)
        n, d = OBJECTS[int(r() * 4)], STATUSES[int(r() * 3)]
        if r() < 0.15:
            y = GENERIC[int(r() * 2)]
        else:
            y = TEMPLATES[int(r() * 3)].format(n=n, e=ACTIONS[d])
        v.append({"condition": (n, d), "response": ["<start>"] + y.split()})
    return v


TRAIN_SET = make_samples(SEED, SAMPLES)
TEST_SET = make_samples(SEED + 500000, TEST)

TABLE = {}
for x in TRAIN_SET:
    y = x["response"]
    for i in range(1, len(y)):
        for a in ((x["condition"], tuple(y[max(0, i - 2):i])), (x["condition"], (y[i - 1],))):
            TABLE.setdefault(a, {})
            TABLE[a][y[i]] = TABLE[a].get(y[i], 0) + 1


def distribution(condition, history):
    d = TABLE.get((condition, tuple(history[-2:]))) or TABLE.get(
        (condition, (history[-1],))) or {".": 1}
    t = sum(d.values())
    return sorted((b, s / t) for b, s in d.items())


def generate(condition, input_len, temperature=0.0, seed=SEED, max_output=12):
    """Step-by-step generation: at each step a distribution is built, a token is
    chosen and appended to the history. The stop condition is the period, the
    upper limit is max_output tokens."""
    r, history, compared = rng(seed + input_len), ["<start>"], 0
    while len(history) - 1 < max_output:
        d = distribution(condition, history)
        compared += input_len + len(history) - 1
        if temperature <= 0:
            choice = max(d, key=lambda bp: (bp[1], bp[0]))[0]
        else:
            a = [(b, p ** (1 / temperature)) for b, p in d]
            u, s = r() * sum(p for _, p in a), 0.0
            choice = a[-1][0]
            for b, p in a:
                s += p
                if u < s:
                    choice = b
                    break
        history.append(choice)
        if choice == ".":
            return history[1:], compared, "condition"
    return history[1:], compared, "limit"


x = TEST_SET[0]
print(f"fictional samples {SAMPLES} responses, test {TEST} tickets")
print(f"ticket condition {x['condition']}, correct action: {ACTIONS[x['condition'][1]]}")
for name, temp, sd in (("greedy decoding", 0.0, SEED), ("greedy decoding", 0.0, SEED + 1),
                       ("sampling 1.0", 1.0, SEED), ("sampling 1.0", 1.0, SEED + 1)):
    y, k, stop = generate(x["condition"], 30, temp, sd, 12)
    print(f"{name:<16} seed {sd}  {len(y):>2} tokens  {stop:<9}  {' '.join(y)}")

fictional samples 600 responses, test 200 tickets
ticket condition ('meter', 'high'), correct action: objection record opened
greedy decoding  seed 20260218   8 tokens  condition  meter record examined and objection record opened .
greedy decoding  seed 20260219   8 tokens  condition  meter record examined and objection record opened .
sampling 1.0     seed 20260218   8 tokens  condition  meter record examined and objection record opened .
sampling 1.0     seed 20260219   7 tokens  condition  meter condition for objection record opened .

The four lines say two things. Greedy decoding gives the exactly same response with both seeds: the seed enters nowhere, because no draw is made at any step. Sampling, on the other hand, gives with the second seed another sentence of 7 tokens instead of 8. Both are correct; changing and being wrong are not the same thing, and this distinction is what the rest of the lesson measures.

Another consequence of the loop shows in latency. The first token of the response is ready after a single step, the last one only after m steps. For this reason the latency of a language model call is not a single number: the time to the first token and the time to the completion of the response behave separately, and the second grows with the output length. This curriculum does not write time in a raw unit but keeps the distinction; a response of eight tokens and a response of eighty tokens are not in the same latency class.

Step-by-step selection also has a structural limit: each step chooses the best for itself and does not look ahead. In the response above, the moment the second token record is chosen, the sentence is bound to the long pattern; had for been chosen at that step, the sentence would have ended two tokens shorter. Greedy decoding is the most likely per step, not per sentence. This means the output length cannot be controlled exactly either, and the limit problem measured by the next section is born from here.

The Cost of Step-by-Step Generation

Each step rereads all of those before it. If a response is m tokens long and the input is n tokens, the total number of comparisons is m·n plus m(m+1)/2: the first term is the repeated reading of the input, the second is the output’s own tail.

INPUT_LEN = 30


def is_correct(y, condition):
    """A response counts only if it is complete and carries the correct object and the correct action."""
    return y[-1] == "." and condition[0] in y and ACTIONS[condition[1]].split()[0] in y


def run(temperature, seed, max_output):
    tokens = compared = hits = limited = 0
    responses = []
    for i, x in enumerate(TEST_SET):
        y, k, stop = generate(x["condition"], INPUT_LEN, temperature, seed + i, max_output)
        tokens += len(y)
        compared += k
        hits += is_correct(y, x["condition"])
        limited += stop == "limit"
        responses.append(tuple(y))
    n = len(TEST_SET)
    return tokens / n, compared / n, hits / n, limited / n, responses


print(f"{'max output':>15} {'tokens':>9} {'comparisons':>14}"
      f" {'at the limit':>15} {'hit rate':>8}")
for mo in (4, 6, 7, 8, 12):
    b, k, i, s, _ = run(0.0, SEED, mo)
    print(f"{mo:>15} {b:>9.2f} {k:>14.1f} {s:>15.4f} {i:>8.4f}")

print(f"\n{'output m':>8} {'comparisons':>14} {'output share':>14}")
for m in (8, 32, 128, 512):
    k = m * INPUT_LEN + m * (m + 1) // 2
    print(f"{m:>8} {k:>14} {m * (m + 1) // 2 / k:>14.4f}")

     max output    tokens    comparisons    at the limit hit rate
              4      4.00          126.0          1.0000   0.0000
              6      5.79          187.7          0.4550   0.5450
              7      6.25          204.0          0.1900   0.8100
              8      6.43          211.1          0.0000   1.0000
             12      6.43          211.1          0.0000   1.0000

output m    comparisons   output share
       8            276         0.1304
      32           1488         0.3548
     128          12096         0.6825
     512         146688         0.8953

The triple is read from here. The bare call is the narrowest setting: maximum output 4 tokens, 126 comparisons per response, hit rate 0.0000. It is cheap and useless, because all of the responses hit the limit and are cut midway. The scaffolding is opening the limit to the real length of the response: at 8 tokens none hits the limit and the hit rate becomes 1.0000. The cost is going from 4.00 to 6.43 tokens and from 126.0 to 211.1 comparisons — about 1.68 times.

The intermediate rows show what kind of failure the limit produces. At six tokens 0.4550 of the responses are cut and the hit rate stays at 0.5450; at seven the share is 0.1900 and the hit rate 0.8100. This failure is silent: a cut response returns no error, it looks like the beginning of a proper sentence. The share hitting the limit is a separate number that has to be measured.

Going from eight to twelve changes nothing: 6.43 and 211.1 stay the same. The maximum output length is a ceiling, not a target; when the stop condition comes earlier, the size of the ceiling creates no cost. This also says how to choose the limit: keeping the ceiling narrow brings no saving, it only creates a risk of being cut. The place to save is the length of the prompt and of the expected response, not the ceiling.

The second table gives where the cost slides in long outputs. With the input fixed at 30 tokens, when the output is 8 tokens the output accounts for only 0.1304 of the comparisons; at 512 tokens this share rises to 0.8953. The cost of a long response depends not on the length of the prompt but on its own length.

The Selection Rule and Temperature

At each step the largest of the distribution can be taken, or the choice can be sampled from the distribution. Temperature is the continuous setting between the two: a small temperature sharpens the distribution, a large temperature flattens it. Zero temperature is the limiting case of sampling and the draw disappears entirely. The sweep below calls the same 200 tickets at seven settings, two runs each; the only thing that changes is the selection rule, and the model and the prompt are the same on every row.

print(f"{'temperature':>11} {'hit 1':>9} {'hit 2':>9} {'changed':>8}"
      f" {'tokens':>9} {'comparisons':>14}")
for temp in (0.0, 0.2, 0.5, 0.8, 1.0, 1.5, 2.0):
    b, k, i1, _, y1 = run(temp, SEED, 16)
    _, _, i2, _, y2 = run(temp, SEED + 1000, 16)
    d = sum(a != b2 for a, b2 in zip(y1, y2)) / len(y1)
    label = f"{temp:.1f}"
    print(f"{label:>11} {i1:>9.4f} {i2:>9.4f} {d:>8.4f} {b:>9.2f} {k:>14.1f}")

temperature     hit 1     hit 2  changed    tokens    comparisons
        0.0    1.0000    1.0000   0.0000      6.43          211.1
        0.2    0.9900    0.9900   0.5550      6.47          212.5
        0.5    0.9500    0.9400   0.6650      6.53          214.3
        0.8    0.8800    0.8750   0.7200      6.50          213.5
        1.0    0.7900    0.8200   0.7650      6.47          212.2
        1.5    0.6250    0.6700   0.8550      6.47          212.1
        2.0    0.5600    0.5600   0.8600      6.46          211.5

The greedy decoding row is the plainest measure of uncertainty in this curriculum: the share of responses that change between two runs is 0.0000. The model is not uncertain here; the uncertainty comes from the selection rule, not from the model itself.

The second row carries the real warning. At temperature 0.2 the hit rate is almost undamaged at 0.9900, but 0.5550 of the responses change between two runs. The same ticket gets two different sentences at two different times, and both are correct. For a product this number is independent design data: if the text of the response is stored somewhere, compared, or matched against an expected output in a test, 0.5550 is a test that breaks on every run.

As the temperature grows the two columns separate. The change between two runs saturates toward 0.8600, while the hit rate falls from 0.9900 to 0.5600. The cost of the scaffolding is paid here in hit rate, and what is received in return is only diversity. The token and comparison columns are almost constant: the cost of temperature is not tokens but accuracy.

It can also be read from the table that three temperatures serve three separate jobs. In a classification or code-assignment task 0.0 is the right setting: repeatable and with the highest hit rate. In a response text, between 0.2 and 0.5 lowers the hit rate very little and takes the text out of monotony. Above 1.0 can be defended only if diversity itself is the purpose of the product, and in this table it costs a hit-rate loss of 0.2100.

The table has one more column that stays silent: the cost of a wrong response. At temperature 1.0, 0.2100 of the responses are wrong, and because these errors are randomly distributed they can be corrected by calling the same prompt again. But calling again is not free — every retry is a new 211.1 comparisons and 6.47 tokens. In a scaffolding where one of five tickets is called a second time, the average cost rises to 1.21 calls, and the hit rate obtained in return is below the already 1.0000 figure of temperature 0.0. Opening up the uncertainty first and then closing it with a retry is a more expensive way of reaching the same number. This is the form the second claim of the curriculum takes in this lesson: the cost of the added scaffolding is certain, its gain is uncertain.

Summary

• Inference is not a single operation but a loop: each step builds a distribution, chooses a token and appends its choice to the end of the input. The tickets and the response samples are fictional, seed 20260218.
• Cost is written in two units. With greedy decoding, 6.43 tokens and 211.1 comparisons per response; with the input fixed at 30 tokens, when the output grows to 512 tokens the output itself accounts for 0.8953 of the comparisons.
• The maximum output length is a source of silent failure: at 4 tokens all responses are cut and the hit rate is 0.0000, at 6 it is 0.5450, at 7 0.8100, at 8 1.0000.
• Greedy decoding gives exactly the same response in two runs (share changed 0.0000). The uncertainty comes from the selection rule, not from the model.
• At temperature 0.2 the hit rate stays at 0.9900 while the share of responses that change between two runs is 0.5550; at 2.0 the hit rate is 0.5600 and the share changed is 0.8600. The token cost is almost constant throughout this sweep.

Next Step

How the response is produced and what the selection rule costs was counted. But in all of these measurements the input was held fixed at 30 tokens. In a real scaffolding the input is not fixed: the ticket stream grows, documents are added, earlier responses are put back. The model also has a window, and that window is fixed. The next lesson measures what happens when the window is exceeded: how many tokens dropping from the head, dropping from the middle and summarizing save on the same task, and how much they lower the hit rate.