Transformer Intuition

The previous lesson split text into tokens and counted that each token has a cost. There is a sequence in hand, but what the model does with it was not asked. In a support stream the topic is stated in the first sentence and referred to in the last sentence only as “this matter”: across the fifty tokens in between, the link has to be carried without breaking.

The structure of the transformer architecture, attention and the embedding layer was built and measured in the Architectures topic of the Deep Learning course; the architecture is not repeated here. This lesson takes up a single question from the user’s side: why can attention use distant context and what does it cost. The answer hides in a single structural detail — in attention the weight of a position is not tied to its distance — and the cost in a single number: the number of comparisons.

• LM9. The ticket streams are fictional: they are generated inside the lesson, seed 20260218. The dataset is 240 streams; each stream is a topic sentence, two to nine filler sentences and a closing question.
• LM10. The tokenizer is the same procedure as in the previous lesson: a vocabulary of pieces and leftmost longest match.
• LM11. The task is a single one: finding the topic object of the stream. The correct answer is the object that appears in the first sentence.
• LM12. In a part of the streams a second object appears near the end. This second object is not the topic; it misleads every procedure that looks at proximity.
• LM13. The match score in the simulator is given by hand: object root 1.00, status root 0.25, the rest 0.02. In a real model this score is learned. What is measured is not where the score comes from but how many positions are compared.
• LM14. Readers differ only in coverage: the fixed window looks at the last k positions, the decaying reader looks at every position but multiplies the weight by a decay with distance, attention looks at every position independently of distance.
• LM15. The measure is the same on every row: topic object hit rate (larger is better) and number of comparisons (smaller is better).
• LM16. The number of comparisons is independent of the run; it is a counted quantity, not a measured one.

Link Distance

The streams vary a single variable in a controlled way: the distance of the topic object from the end of the sequence.

# SIMULATOR — the model is a simulator written in the lesson with the standard
# library; no real endpoint is called. Ticket streams are FICTIONAL, seed 20260218.
SEED, M32, STREAMS = 20260218, 0xFFFFFFFF, 240

ROOTS = """meter seal valve line invoice tariff reading payment customer address fault code
service team visit record ticket support repair day month year hour number detail process
system error result value late objection application reply wait send back open close read
write find look come give do want work broken cracked missing normal high low new old
closed please hello thanks good request again still status matter week same two every
problem with for and a the this that what is of our no there but did got wrote opened
came applied asked waited through nothing answer too also as well about i ago""".split()
SUFFIXES = """s es ed ing er ly""".split()
PIECES = set(ROOTS) | set(SUFFIXES) | set("abcdefghijklmnopqrstuvwxyz0123456789")


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(text):
    out, cur = [], ""
    for c in text.lower() + " ":
        if c.isalnum():
            cur += c
            continue
        i = 0
        while i < len(cur):
            for j in range(min(len(cur), i + 9), i, -1):
                if cur[i:j] in PIECES:
                    out.append(cur[i:j])
                    i = j
                    break
            else:
                out.append(cur[i])
                i += 1
        cur = ""
        if c in ".,?":
            out.append(c)
    return out


OBJECTS = {"invoice": ("our invoice", ["high", "missing"]),
           "tariff": ("our tariff", ["high", "old"]),
           "reading": ("our reading", ["high", "low"]),
           "meter": ("our meter", ["broken", "old"]),
           "seal": ("our seal", ["cracked", "missing"]),
           "valve": ("our valve", ["closed", "broken"]),
           "line": ("our line", ["closed", "old"]),
           "payment": ("our payment", ["missing", "late"])}
FILLERS = ["I wrote again about this but got no reply.",
           "I still waited and there is no result.",
           "I opened a ticket this month too.",
           "The service team came but did nothing.",
           "There is no result with the record number.",
           "I applied through the system but no reply came.",
           "I wrote again two days ago.",
           "I asked for details about the same status every week."]
OBJECT_NAMES = list(OBJECTS)

DATA = []
for i in range(STREAMS):
    r = rng(SEED + i)
    n = OBJECT_NAMES[int(r() * 8)]
    long_name, statuses = OBJECTS[n]
    lines = [f"Hello, the status of {long_name} is {statuses[int(r() * 2)]}."]
    j = int(r() * 8)
    for _ in range(2 + int(r() * 8)):
        lines.append(FILLERS[j % 8])
        j += 3
    distractor = None
    if r() < 0.35:
        distractor = OBJECT_NAMES[(OBJECT_NAMES.index(n) + 1 + int(r() * 7)) % 8]
        lines.append(f"There is also a record for the {distractor}.")
    lines.append("What is the status of this matter?")
    seq = tokenize(" ".join(lines))
    DATA.append({"seq": seq, "object": n, "distractor": distractor,
                 "distance": len(seq) - 1 - seq.index(n)})

dist = [x["distance"] for x in DATA]
print(f"{STREAMS} fictional ticket streams; {sum(x['distractor'] is not None for x in DATA)}"
      f" streams contain a second object")
print(f"stream length {min(len(x['seq']) for x in DATA)}"
      f"-{max(len(x['seq']) for x in DATA)} tokens, mean"
      f" {sum(len(x['seq']) for x in DATA) / STREAMS:.1f}")
print(f"distance of the topic object from the end {min(dist)}-{max(dist)}, mean {sum(dist) / STREAMS:.1f}")
print("\n" + " ".join(DATA[3]["seq"]))

240 fictional ticket streams; 83 streams contain a second object
stream length 34-113 tokens, mean 70.9
distance of the topic object from the end 27-106, mean 63.9

hello , the status of our reading is low . i wrote again two day s ago . i still waited and there is no result . there is no result with the record number . what is the status of this matter ?

The closing question is the same in every stream and carries no information by itself: the answer sits 63.9 tokens behind on average. In about three of every eight streams a second object appears near the end, and it is a trap for any procedure that looks at proximity.

This setup is not an exercise but an ordinary production situation. To route a support stream to the right team the topic object is needed, and the newest sentence in the stream almost never carries it. The first path that comes to mind is the cheapest one: look at the last few sentences, throw the rest away. The table below gives the number for that path, and what attention solves is seen only when it is read against that number.

The Reader That Looks at Distance and the Reader That Does Not

Three readers use the same match score and differ only in which positions they compare. The fixed window looks at the last k positions; the decaying reader looks at every position but multiplies the weight by 0.94 at every token; attention looks at every position independently of distance.

OBJECT_SET = set(OBJECT_NAMES)
STATUS_SET = {d for _, ds in OBJECTS.values() for d in ds}


def match_score(t):
    """Match of the query with the position. Learned in a real model; given by hand here."""
    return 1.0 if t in OBJECT_SET else (0.25 if t in STATUS_SET else 0.02)


def read(seq, window=None, decay=1.0):
    n = len(seq)
    start = 0 if window is None else max(0, n - window)
    best, best_p = None, -1.0
    for i in range(start, n):
        p = match_score(seq[i]) * (decay ** (n - 1 - i))
        if p > best_p:
            best_p, best = p, seq[i]
    return (best if best in OBJECT_SET else None), n - start


BUCKETS = [(0, 32), (33, 64), (65, 999)]


def measure(name, **kw):
    correct, total, compared = [0, 0, 0], [0, 0, 0], 0
    for x in DATA:
        c, m = read(x["seq"], **kw)
        compared += m
        j = next(i for i, (a, b) in enumerate(BUCKETS) if a <= x["distance"] <= b)
        total[j] += 1
        correct[j] += c == x["object"]
    print(f"{name:<26} {compared / STREAMS:>8.1f} {sum(correct) / STREAMS:>7.4f} "
          + " ".join(f"{d / t:>7.4f}" for d, t in zip(correct, total)))


print(f"{'reader':<26} {'compared':>8} {'hit':>7} "
      + " ".join(f"{a}-{b:<3}".rjust(7) for a, b in BUCKETS))
for k in (8, 16, 32, 64):
    measure(f"last {k} tokens", window=k)
measure("decay with distance", decay=0.94)
measure("full attention")
print("streams per bucket" + "".join(
    f"{sum(1 for x in DATA if a <= x['distance'] <= b):>8}" for a, b in BUCKETS).rjust(44))
y = [read(x["seq"], window=16)[0] for x in DATA]
print(f"last 16 tokens: {sum(c == DATA[i]['object'] for i, c in enumerate(y))} correct,"
      f" {sum(c is not None and c != DATA[i]['object'] for i, c in enumerate(y))} wrong object,"
      f" {sum(c is None for c in y)} empty response")

reader                     compared     hit   0-32   33-64   65-999
last 8 tokens                   8.0  0.0000  0.0000  0.0000  0.0000
last 16 tokens                 16.0  0.0000  0.0000  0.0000  0.0000
last 32 tokens                 32.0  0.0833  1.0000  0.0000  0.0000
last 64 tokens                 57.9  0.4917  1.0000  0.9333  0.0000
decay with distance            70.9  0.3333  1.0000  0.5714  0.0000
full attention                 70.9  1.0000  1.0000  1.0000  1.0000
streams per bucket                          20     105     115
last 16 tokens: 0 correct, 83 wrong object, 157 empty response

The triple is read from here. The bare call is the cheapest reading: the last 16 tokens, 16 comparisons per stream, hit rate 0.0000. The last line opens up that zero — in 83 streams the second object that enters the window is returned as the wrong answer, in 157 streams an empty response is given because no object is found in the window. The cheap reading has two separate failure modes, and one is silent, the other is not.

As the window grows the hit rate rises step by step but is always cut at the same place: the last 32 tokens give 1.0000 in the 0-32 bucket and exactly zero beyond it; the last 64 tokens rise to 0.9333 in the 33-64 bucket and again give zero beyond 65 tokens. The window is good where it sees and nothing where it does not.

The decaying reader is a more interesting failure. It looks at every position, so its coverage is complete, but it stays at 0.3333. The distance penalty, 0.94 to the power of sixty, drops to about 0.024, and the 1.00 match of the distant object falls below 0.25 of a nearby status root. Coverage is not enough; the weight must be independent of distance. This is exactly what attention does, and the 1.0000 hit rate comes from here.

The Growth Rate of the Cost

A weight independent of distance is not free. For a single query, a reading costs n comparisons; but the model does this for every position. In a causal pass the i-th position compares the i positions before it, and the total is n(n+1)/2.

print(f"{'context n':>9} {'full attention':>14} {'window 64':>12} {'ratio':>7} {'growth':>7}")
previous = None
for n in (64, 128, 256, 512, 1024):
    full, win = n * (n + 1) // 2, n * 64 - 64 * 63 // 2
    growth = "-" if previous is None else f"{full / previous:.2f}"
    print(f"{n:>9} {full:>14} {win:>12} {full / win:>7.2f} {growth:>7}")
    previous = full


def sample(seq, seed):
    r = rng(seed + len(seq))
    scores = [match_score(t) for t in seq]
    u, s = r() * sum(scores), 0.0
    for t, w in zip(seq, scores):
        s += w
        if u < s:
            return t if t in OBJECT_SET else None
    return None


print(f"\ngreedy decoding   hit {sum(read(x['seq'])[0] == x['object'] for x in DATA) / STREAMS:.4f}"
      "   (same in both runs)")
for t in (SEED, SEED + 1):
    d = sum(sample(x["seq"], t) == x["object"] for x in DATA) / STREAMS
    print(f"sampling seed {t}   hit {d:.4f}")

context n full attention    window 64   ratio  growth
       64           2080         2080    1.00       -
      128           8256         6176    1.34    3.97
      256          32896        14368    2.29    3.98
      512         131328        30752    4.27    3.99
     1024         524800        63520    8.26    4.00

greedy decoding   hit 1.0000   (same in both runs)
sampling seed 20260218   hit 0.4292
sampling seed 20260219   hit 0.4042

The growth column is a one-sentence rule: when the context doubles, the comparisons quadruple. The fixed window grows linearly, and the ratio between them reaches 8.26 at 1024 tokens. The context window discussion of the next lesson and the whole cost calculation of this curriculum pass between these two curves: why the limit is a number and why it is not enlarged generously is seen from here.

The counterpart of this table on the user’s side is this: every token added to the context writes a faster-than-linear cost, because it will be compared with all the ones that follow. A document added to the prompt “to be safe” does not pay its cost only with its own length. Latency follows the same curve and is referred to in this curriculum as a class: single call, chain, parallel. A single call with a long context can be more expensive than two calls with short contexts, and which to choose is not known without measuring.

This lesson measured not what attention is but what it buys. Attention does not explain what the model knows; it only determines which positions in the sequence can be linked to each other. Where the match scores come from is still a gap.

The last three lines are this lesson’s measure of uncertainty. The number of comparisons is a counted quantity and is independent of the run; the reading decision is not. When the largest of the weights is taken, the hit rate is 1.0000 in both runs. When the choice is made from the same weights by sampling, the same input gives 0.4292 and 0.4042: two runs, two numbers, the same scaffolding. The selection rule itself is a design decision, and the next lesson measures it.

Summary

• The distinguishing feature of attention is not coverage but the weight being independent of distance. The reader with complete coverage but decay with distance stays at 0.3333, while attention gives 1.0000.
• Fixed-window reading is good where it sees and nothing where it does not: the last 32 tokens give 1.0000 in the near bucket and 0.0000 beyond it; the last 64 tokens give an overall hit rate of 0.4917.
• The cheap reading has two failure modes: in 83 streams the nearby second object is returned as the wrong answer, in 157 streams an empty response comes back.
• The cost is the number of comparisons and it grows quadratically: when the context doubles the comparisons quadruple, and at 1024 tokens it is 8.26 times the fixed window.
• From the same weights, greedy decoding gives 1.0000 in both runs, while sampling gives 0.4292 and 0.4042.

Next Step

The model can use distant context and the cost of that was counted. But where did this ability come from: the match scores were written by hand in this lesson, and in a real model they are learned. Learning has two separate stages and the two solve separate problems. The next lesson splits a task set in two and counts how many examples it takes to improve each: which question is answered from the broad text corpus, which from the field’s own examples.