Pretraining and Fine-Tuning

The previous lesson showed that attention can use distant context and counted its cost, but it wrote the match scores by hand. In a real model those scores are learned. Learning is not a single event: it solves two separate problems, in two separate stages, with two separate data sets. This lesson treats the two stages not as a procedure but as which problem is solved in which one.

The procedure of transfer learning and pretrained language models was measured in the Architectures topic of the Deep Learning course and in the Natural Language Processing and Computer Vision course; it is not repeated here. The question here belongs to the user: there is a task in hand, which stage’s product solves it, and if it does not, how many examples are needed. The measure is the number of examples.

• LM17. The general-text corpus and the support tickets are fictional: they are generated inside the lesson, seed 20260218. The corpus is 1,600 sentences, the ticket set 600 records.
• LM18. The model is a simulator: pretraining and fine-tuning are the same procedure — counting what follows the last one and the last two tokens. The only thing that differs is which text is counted.
• LM19. Pretraining is unlabeled: it sees only text and receives no task label. Fine-tuning is done with labeled domain examples and its cost is the number of those examples.
• LM20. The fine-tuning table is queried first, the pretraining table as a fallback. If a context has been seen once in fine-tuning, it suppresses the knowledge of pretraining in that context.
• LM21. The first task is next-token hit rate and it is measured on two separate texts: general text and domain text.
• LM22. The second task is the organization’s own mapping: from ticket to fault code — a 12-class, fictional and arbitrary code derived from the object and the status. This code never appears in general text.
• LM23. The classifier relies on independent token counts and is built only from fine-tuning examples; at zero examples it has nothing in hand.
• LM24. The criterion is the same on every row: hit rate on the held-out share (larger is better) and number of labeled examples (smaller is better).
• LM25. The measure of uncertainty is which ones the fine-tuning examples are: the same number of examples is selected with a second seed and the second run is printed.

Two Texts, Two Tasks

The general-text corpus consists of ordinary sentences; the tickets use the same grammar but carry the objects and statuses of the domain. The fault code exists only in the tickets.

# SIMULATOR — the model is a simulator written in the lesson with the standard
# library; no real endpoint is called. The general-text corpus and the support
# tickets are FICTIONAL, seed 20260218.
SEED, M32, GENERAL, TICKETS = 20260218, 0xFFFFFFFF, 1600, 600


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


SUBJECTS = ["i", "we", "you", "team", "service", "customer", "system", "clerk"]
VERBS = ["came", "wrote", "waited", "sent", "opened", "gave", "made", "saw"]
NOUNS = ["record", "ticket", "detail", "reply", "result", "number", "address", "process"]
ADJECTIVES = ["new", "old", "open", "closed", "normal", "missing"]
GENERAL_TEMPLATES = ["{o} {f} a {a} .", "{o} {f} a {b} for {a} .",
                     "this {s} {a} and {o} {f} .", "{o} {f} a {s} {a} .",
                     "{s} {a} but even so {o} {f} ."]

CORPUS = []
for i in range(GENERAL):
    r = rng(SEED + i)
    CORPUS.append(GENERAL_TEMPLATES[int(r() * 5)].format(
        o=SUBJECTS[int(r() * 8)], a=NOUNS[int(r() * 8)], b=NOUNS[int(r() * 8)],
        s=ADJECTIVES[int(r() * 6)], f=VERBS[int(r() * 8)]).split())

OBJECTS = ["invoice", "tariff", "meter", "valve"]
STATUSES = ["high", "missing", "broken"]
CODES = {(n, d): f"{'ABCD'[i]}{j + 1}" for i, n in enumerate(OBJECTS)
         for j, d in enumerate(STATUSES)}
TICKET_TEMPLATES = ["{o} {f} a record for {n} , state {d} .",
                    "this {n} {d} but even so {o} {f} .",
                    "{o} {f} that {n} status is {d} .",
                    "{n} {d} and {o} {f} the reply ."]

DATA = []
for i in range(TICKETS):
    r = rng(SEED + 100000 + i)
    n, d = OBJECTS[int(r() * 4)], STATUSES[int(r() * 3)]
    seq = TICKET_TEMPLATES[int(r() * 4)].format(
        n=n, d=d, o=SUBJECTS[int(r() * 8)], f=VERBS[int(r() * 8)]).split()
    DATA.append({"seq": seq, "code": CODES[(n, d)]})

C_TRAIN, C_HOLD = CORPUS[:1400], CORPUS[1400:]
TRAIN, HOLD = DATA[:400], DATA[400:]
codes = sorted({x["code"] for x in DATA})
print(f"general-text corpus {len(CORPUS)} sentences ({len(C_TRAIN)} training, {len(C_HOLD)} held out)")
print(f"support tickets {len(DATA)} ({len(TRAIN)} fine-tuning pool, {len(HOLD)} held out)")
print(f"fault code {len(codes)} classes: {' '.join(codes)}")
print(f"\ngeneral: {' '.join(C_TRAIN[0])}")
print(f"ticket:  {' '.join(TRAIN[0]['seq'])}   -> {TRAIN[0]['code']}")
shared = {t for c in C_TRAIN for t in c} & {t for x in DATA for t in x["seq"]}
print(f"corpus vocabulary {len({t for c in C_TRAIN for t in c})}, ticket vocabulary"
      f" {len({t for x in DATA for t in x['seq']})}, shared {len(shared)}")

general-text corpus 1600 sentences (1400 training, 200 held out)
support tickets 600 (400 fine-tuning pool, 200 held out)
fault code 12 classes: A1 A2 A3 B1 B2 B3 C1 C2 C3 D1 D2 D3

general: old address but even so service saw .
ticket:  you wrote a record for meter , state missing .   -> C2
corpus vocabulary 38, ticket vocabulary 39, shared 27

Of the 39 tokens in the ticket vocabulary, 27 also appear in the general corpus. What separates a ticket is not the grammar but the four objects and three statuses. The fault code, on the other hand, is written in no text: it is the organization’s own mapping and can be learned only from the label.

This distinction is not a made-up arrangement but the carrier of the measurement. The answer to a task is either somewhere in the text of the world — how a sentence is built, which word follows which — or it lives only in that organization’s records. The first comes without asking for any examples, the second makes you pay for every example one by one. The two tables below test the same simulator with the same criterion on two classes of task and write where the difference comes from in terms of the number of examples.

The Product of Pretraining

Pretraining is unlabeled: the simulator only reads the corpus and counts which token follows which. Fine-tuning applies the same procedure to the domain text. The three columns give the same criterion under three separate conditions.

HOLD_SEQS = [x["seq"] for x in HOLD]


def train(sentences):
    """Pretraining and fine-tuning are the same procedure: counting what follows
    the last one and the last two tokens. Only the text differs."""
    t = {}
    for c in sentences:
        for i in range(1, len(c)):
            for a in [(c[i - 1],)] + ([(c[i - 2], c[i - 1])] if i > 1 else []):
                t.setdefault(a, {})
                t[a][c[i]] = t[a].get(c[i], 0) + 1
    return t


def predict(history, tuned, pretrained, default):
    """Fine-tuning table first, pretraining table as a fallback."""
    for t in (tuned, pretrained):
        for a in (tuple(history[-2:]), tuple(history[-1:])):
            d = t.get(a)
            if d:
                return max(sorted(d), key=d.get)
    return default


def next_hit_rate(sentences, tuned, pretrained, default):
    correct = total = 0
    for c in sentences:
        for i in range(1, len(c)):
            total += 1
            correct += predict(c[:i], tuned, pretrained, default) == c[i]
    return correct / total


PRETRAINED = train(C_TRAIN)
DEFAULT = max(sorted({t for c in C_TRAIN for t in c}),
              key=lambda t: sum(c.count(t) for c in C_TRAIN))
K_LIST = [0, 2, 4, 8, 16, 32, 64, 128, 256, 400]
print(f"{'examples':>8} {'general text':>12} {'domain text':>11} {'untrained domain':>16}")
for k in K_LIST:
    tuned = train([x["seq"] for x in TRAIN[:k]]) if k else {}
    print(f"{k:>8} {next_hit_rate(C_HOLD, tuned, PRETRAINED, DEFAULT):>12.4f}"
          f" {next_hit_rate(HOLD_SEQS, tuned, PRETRAINED, DEFAULT):>11.4f}"
          f" {next_hit_rate(HOLD_SEQS, tuned, {}, DEFAULT):>16.4f}")

examples general text domain text untrained domain
       0       0.3186      0.1983           0.1296
       2       0.2974      0.3305           0.2528
       4       0.2992      0.4439           0.3733
       8       0.2913      0.4783           0.4102
      16       0.2886      0.5360           0.5301
      32       0.2727      0.5347           0.5347
      64       0.2595      0.5476           0.5476
     128       0.2736      0.5502           0.5502
     256       0.2736      0.5256           0.5256
     400       0.2692      0.5437           0.5437

The first row gives the product of pretraining alone. The simulator that has seen no domain example gets 0.1983 on domain text; the same measure without pretraining is 0.1296. This is transfer, and it is a number: the 27 shared tokens make the domain’s own text predicted with a clearly higher hit rate. A domain example is not free; pretraining has paid a part of it in advance.

The third column becomes equal to the second from 32 examples. When enough domain examples have accumulated, the contribution of pretraining falls to zero: the fallback table is never reached again. The value of transfer exists while examples are few. This also says where to look when measuring — in a setup with few examples the difference between a pretrained and an untrained run is worth measuring, in a setup with many examples that difference is zero even if it is measured.

The first column, on the other hand, moves in the opposite direction: starting from 0.3186, it falls to 0.2595 at 64 examples, then recovers to 0.2692 at 400. The loss is real and its cause is the order of the tables — a context seen once in fine-tuning suppresses the knowledge of pretraining in that context. While examples are few, this single observation produces an unrepresentative preference; as examples increase the preference approaches the general distribution but does not return to the starting level. The cost of fine-tuning is not only labeled examples but a loss of 0.0494 in general ability.

What Depends on the Label

The second task is the organization’s own mapping. There is no such thing as a fault code in general text; however much text is read, this mapping cannot be derived.

import math


def train_classifier(examples):
    count, total = {}, {}
    for x in examples:
        k = x["code"]
        total[k] = total.get(k, 0) + 1
        for t in set(x["seq"]):
            count[(t, k)] = count.get((t, k), 0) + 1
    return count, total


def classify(model, seq):
    count, total = model
    if not total:
        return codes[0]
    best, best_p = None, None
    for k in sorted(total):
        p = math.log(total[k])
        for t in sorted(set(seq)):
            p += math.log((count.get((t, k), 0) + 1) / (total[k] + 2))
        if best_p is None or p > best_p:
            best, best_p = k, p
    return best


def pick_sample(pool, k, seed):
    r, p = rng(seed), list(pool)
    for i in range(len(p) - 1, 0, -1):
        j = int(r() * (i + 1))
        p[i], p[j] = p[j], p[i]
    return p[:k]


def code_hit_rate(examples):
    m = train_classifier(examples)
    return sum(classify(m, x["seq"]) == x["code"] for x in HOLD) / len(HOLD)


print(f"{'examples':>8} {'code hit':>12} {'seen classes':>12} {'second run':>13}")
threshold = None
for k in K_LIST:
    o = pick_sample(TRAIN, k, SEED)
    i1, i2 = code_hit_rate(o), code_hit_rate(pick_sample(TRAIN, k, SEED + 1))
    print(f"{k:>8} {i1:>12.4f} {len({x['code'] for x in o}):>12} {i2:>13.4f}")
    if threshold is None and i1 >= 0.90:
        threshold = k
print(f"\nsmallest number of examples passing the 0.90 threshold: {threshold}")
print(f"share of the most frequent code in the held-out part:"
      f" {max(sum(x['code'] == k for x in HOLD) for k in codes) / len(HOLD):.4f}")

examples     code hit seen classes    second run
       0       0.0900            0        0.0900
       2       0.1200            2        0.0800
       4       0.2000            3        0.0950
       8       0.2150            6        0.2000
      16       0.1600            8        0.2450
      32       0.2450           11        0.3600
      64       0.4800           12        0.4850
     128       0.5700           12        0.6550
     256       0.8400           12        0.8350
     400       0.9000           12        0.9000

smallest number of examples passing the 0.90 threshold: 400
share of the most frequent code in the held-out part: 0.1200

At zero examples the hit rate is 0.0900 and it is below the 0.1200 of the procedure that always says the most frequent code. The contribution of pretraining to this task is not zero but negative: the model knows the language, not the mapping. The seen-classes column gives the mechanical reason — all twelve codes enter the pool only at 64 examples, and a code that has not been seen before cannot be said under any condition.

The threshold is 400 examples. Same task, same model, same criterion: the equivalent of pretraining is zero, that of fine-tuning is 400 examples. This is the question that splits a task set in two — whether the answer is written in the text or lives only in the organization’s records.

The curve is not flat either. The hit rate that rises to 0.2150 at eight examples goes down to 0.1600 at sixteen: because the newly arriving classes dilute the existing counts, the number regresses in an intermediate region. Reading an example curve from two points is therefore misleading; to be able to say which number of examples is enough, one has to see where the curve flattens. Here it has not flattened yet: the last 144 examples, from 256 to 400, still add 0.0600 to the hit rate, and the threshold is passed exactly at the last row of the pool.

The last column shows the uncertainty. When the number of examples is held fixed and which examples are drawn is changed, 128 gives 0.5700 and 0.6550, 32 gives 0.2450 and 0.3600. Both are numbers of the same setup. An example curve read from a single run hides the variability that comes from the selection of examples.

The counterpart of these two stages on the user’s side is not equal. Pretraining is closed inside a component given in this curriculum: it has been done, it cannot be changed, its number can be read only from the surface. Fine-tuning is a decision, and its cost is paid up front — the examples to be labeled, the person who will give the label, the consistency of the label. The table above is the price tag of that decision: 400 examples for 0.90 in this task. The question to ask when deciding is not “can the model do this” but how many examples it costs to reach this number. Mechanisms of the later courses, such as prompt design and grounding with retrieval, are ways of approaching the same number without paying for labels, and all of them are read against this table.

Summary

• Pretraining and fine-tuning are the same procedure in this simulator; the only thing that differs is which text is counted. The texts and the fault code are fictional, seed 20260218.
• The contribution of pretraining to domain text is measurable: 0.1983 without seeing a single domain example, 0.1296 without pretraining. The contribution exists while examples are few and falls to zero from 32 examples.
• The cost of fine-tuning is not only 400 labeled examples: the general-text hit rate falls from 0.3186 to 0.2692, and the lowest point is at 64 examples, at 0.2595.
• The organization’s own mapping is never solved by pretraining: a hit rate of 0.0900 is below the 0.1200 share of the most frequent code. The 0.90 threshold is passed only with 400 labeled examples.
• When the same number of examples is selected with another seed, the number moves: at 128 examples, 0.5700 and 0.6550.

Next Step

Where the model learns what from was split, and the cost of each stage was written in numbers of examples. But in this lesson the model always produced a single prediction: the most likely token, in a single step. A real response is not a single token; it is produced step by step and every step is a choice. The next lesson measures that choice: the output length against the number of tokens produced, greedy decoding against sampling in two runs on the same prompt, and where the distribution temperature stands between the two.