Pretraining and Fine-Tuning

In the previous lesson the bag of words passed the domain baseline by 0.5200, but 1,100 labeled notes were spent for it. Every label is the counterpart of the time an inspector spent in the field, and there can be twenty of them at hand. Unlabeled notes, on the other hand, are plentiful: the measurement network produces thousands of notes every period and none of them has an event kind written on it.

The question of this lesson is this: can a proxy task be built from this unlabeled pile, and when the representation learned there is carried over to the labeled task, with how many labels does it add how much of a difference. The proxy task is masked prediction: one word of a note is hidden and predicted from the remaining words. The correct answer is in the text itself, and no label from outside is needed. The attention layer, multi-head attention and positional encoding were built and measured in the Deep Learning course, architectures/06–07; what is measured here is not the architecture. What is measured is the loop: pretraining first, then fine-tuning.

• NL48. The text fiction is the same as that of the previous lesson and is fiction; seed 20260218.
• NL49. The set is divided into three: the first 700 notes are the unlabeled pool (only their text is used, the event kind is never read), the next 400 notes are the labeled pool, the last 500 notes are the held-out set. The held-out set is the same 500 notes as in the previous lesson.
• NL50. The vocabulary is built from the unlabeled pool only; words that occur in at least two notes are taken.
• NL51. The model is written within the lesson: an embedding table of six numbers per word, the average of the embeddings of the note, and a four-class linear body. The table has 654 parameters, the body 28.
• NL52. The baseline of the proxy task is the most frequent word: the procedure that writes the most frequent word of the vocabulary at every masked position.
• NL53. Four setups are compared: from scratch (the table is initialized randomly and learned), random table, frozen (not learned — the control setup), pretrained and frozen, pretrained and fine-tuned.
• NL54. The number of labeled examples is measured at five values: 10, 20, 40, 80, 160. The held-out set is the same in all five.
• NL55. Every number is printed with two runs; the runs differ in the initialization seed of the body.

The Unlabeled Pile and Three Separations

The only thing pretraining requires is the text itself. For this reason the separation here is threefold: a pool where the label is never read, a pool where the label is spent by counting, and a held-out set that none of them touches.

# FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
KINDS = ("ordinary", "unreachable", "meter fault", "illicit use")
SHARES = (0.46, 0.22, 0.19, 0.13)
PHRASES = {t: s.split("|") for t, s in zip(KINDS, (
    "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|"
    "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|"
    "gosterge okunakli|gostergede rakam net",
    "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|"
    "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|"
    "bodrum kilitli|adres bulunamadi|adres eksik",
    "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|"
    "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|"
    "mekanizmada takilma|sayac durmus|sayac calismiyor",
    "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|"
    "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|"
    "sayac ters takilmis|sayaci ters takmislar"))}
NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|"
         "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|")
REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|")
SUSPICION_P = {"illicit use": 0.86, "meter fault": 0.21, "unreachable": 0.07, "ordinary": 0.03}


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_note(n):
    r = rng(SEED + 41000 + n)
    u, k = r(), 0.0
    for i, p in enumerate(SHARES):
        k += p
        if u < k:
            t = KINDS[i]
            break
    p = [PHRASES[t][int(r() * 12)]]
    if r() < 0.55:
        p.append(PHRASES[t][int(r() * 12)])
    if r() < 0.34:                          # side record borrowed from another kind
        p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)])
    if r() < 0.62:
        p.append(NOISE[int(r() * 8)])
    if r() < 0.30:
        p.append(REGIONS[int(r() * 5)])
    return {"text": ", ".join(p), "kind": t, "suspicious": 1 if r() < SUSPICION_P[t] else 0}


NOTES = [build_note(i) for i in range(1600)]
KIND_INDEX = {t: i for i, t in enumerate(KINDS)}
POOL, LABELED, HELD_OUT = NOTES[:700], NOTES[700:1100], NOTES[1100:]


def split(m):
    return [w.strip(",").lower() for w in m.split() if w.strip(",")]


d = {}
for x in POOL:
    for w in set(split(x["text"])):
        d[w] = d.get(w, 0) + 1
S = {w: i for i, w in enumerate(sorted(w for w, c in d.items() if c >= 2))}
V, EMB = len(S), 6
SEQUENCES = [z for z in ([S[w] for w in split(x["text"]) if w in S] for x in POOL)
             if len(z) >= 3]
counts = [0] * V
for z in SEQUENCES:
    for i in z:
        counts[i] += 1
MOST_COMMON_WORD = max(range(V), key=lambda i: counts[i])
kind_counts = {}
for x in LABELED[:160]:
    kind_counts[x["kind"]] = kind_counts.get(x["kind"], 0) + 1
MOST_COMMON = max(kind_counts, key=kind_counts.get)
BASELINE = sum(x["kind"] == MOST_COMMON for x in HELD_OUT) / len(HELD_OUT)
print(f"unlabeled pool {len(SEQUENCES)} notes, labeled pool {len(LABELED)}, "
      f"held-out {len(HELD_OUT)}")
print(f"vocabulary {V}, embedding size {EMB}, table parameters {V * EMB}, "
      f"body parameters {4 * EMB + 4}")
print(f"domain baseline (most frequent class '{MOST_COMMON}') {BASELINE:.4f}")

unlabeled pool 664 notes, labeled pool 400, held-out 500
vocabulary 109, embedding size 6, table parameters 654, body parameters 28
domain baseline (most frequent class 'ordinary') 0.4420

The parameter counts show the tension of this lesson: the table has 654, the body 28. Learning all 682 parameters with ten labeled examples is a situation where the capacity exceeds the data many times over, and the Introduction to Machine Learning course, generalization/02, had measured this. The claim of pretraining is this: 654 parameters can be learned from unlabeled text, which leaves 28 parameters to be learned.

Proxy Task: Masked Prediction

In masked prediction every position of every note is hidden in turn, and the hidden word is predicted from the average of the embeddings of the remaining words. What is learned is two tables: the input embedding and the output embedding. Only the first is carried over to the real task.

def make_table(seed, scale=0.5):
    r = rng(seed)
    return [[(r() + r() + r() - 1.5) * scale for _ in range(EMB)] for _ in range(V)]


def train_masked(seed, epochs=4, step=0.1):     # THE MODEL: proxy task
    T, U, c = make_table(seed), make_table(seed + 7, 0.3), [0.0] * V
    for _ in range(epochs):
        for z in SEQUENCES:
            for k in range(len(z)):
                target, context = z[k], z[:k] + z[k + 1:]
                h = [sum(T[i][j] for i in context) / len(context) for j in range(EMB)]
                o = [sum(U[i][j] * h[j] for j in range(EMB)) + c[i] for i in range(V)]
                e = [math.exp(p - max(o)) for p in o]
                s = sum(e)
                dh = [0.0] * EMB
                for i in range(V):
                    f = (e[i] / s - (1.0 if i == target else 0.0)) * step
                    c[i] -= f
                    for j in range(EMB):
                        dh[j] += f * U[i][j]
                        U[i][j] -= f * h[j]
                for i in context:
                    for j in range(EMB):
                        T[i][j] -= dh[j] / len(context)
    return T, U, c


def score_masked(T, U, c):
    hit = base = total = 0
    for z in SEQUENCES[:300]:
        for k in range(len(z)):
            target, context = z[k], z[:k] + z[k + 1:]
            h = [sum(T[i][j] for i in context) / len(context) for j in range(EMB)]
            o = [sum(U[i][j] * h[j] for j in range(EMB)) + c[i] for i in range(V)]
            hit += max(range(V), key=lambda i: o[i]) == target
            base += MOST_COMMON_WORD == target
            total += 1
    return hit / total, base / total


T0, U0, c0 = train_masked(SEED)
mk, mt = score_masked(T0, U0, c0)
print(f"masked prediction: baseline (most frequent word) {mt:.4f}, proxy task {mk:.4f}, "
      f"difference {mk - mt:+.4f}")
print(f"labels spent 0; pretraining {4} epochs, {len(SEQUENCES)} notes")

masked prediction: baseline (most frequent word) 0.0419, proxy task 0.3450, difference +0.3032
labels spent 0; pretraining 4 epochs, 664 notes

On its own measure the proxy task passes its baseline by 0.3032, and the number of labels spent is zero. The Introduction to Machine Learning course, fundamental-concepts/03, had built the same structure with a numerical proxy task, and there too the proxy task had passed its own baseline — but when the contribution of the carried-over representation was measured, the share of the learned part had come out minus 0.0027. The result of that lesson was this: passing the baseline on the proxy task does not bring passing the baseline on the real task. The same question is asked again here and its answer is measured again.

The Difference It Adds and With How Many Labels

The real task is the task of the previous lesson: the event kind of the note. The only thing that changes is how many labels are at hand. The control setup in the fourth line is the real instrument of this section: if a randomly initialized and never trained table is frozen and used with the same body, how much of the number comes from averaging itself.

def encode(x):
    return [S[w] for w in split(x["text"]) if w in S] or [MOST_COMMON_WORD]


LABELED_ENC = [(encode(x), KIND_INDEX[x["kind"]]) for x in LABELED]
HELD_ENC = [(encode(x), KIND_INDEX[x["kind"]]) for x in HELD_OUT]


def train(T, D, seed, learn, epochs=150, step=3.0):    # THE MODEL
    T = [u[:] for u in T]
    r = rng(seed)
    W = [[(r() - 0.5) * 0.2 for _ in range(EMB)] for _ in range(4)]
    b, n = [0.0] * 4, len(D)
    for _ in range(epochs):
        gW = [[0.0] * EMB for _ in range(4)]
        gb = [0.0] * 4
        gT = {} if learn else None
        for z, y in D:
            h = [sum(T[i][j] for i in z) / len(z) for j in range(EMB)]
            o = [sum(W[q][j] * h[j] for j in range(EMB)) + b[q] for q in range(4)]
            e = [math.exp(p - max(o)) for p in o]
            s = sum(e)
            dh = [0.0] * EMB
            for q in range(4):
                f = e[q] / s - (1.0 if q == y else 0.0)
                gb[q] += f
                for j in range(EMB):
                    gW[q][j] += f * h[j]
                    dh[j] += f * W[q][j]
            if learn:
                for i in z:
                    g = gT.setdefault(i, [0.0] * EMB)
                    for j in range(EMB):
                        g[j] += dh[j] / len(z)
        for q in range(4):
            b[q] -= step * gb[q] / n
            for j in range(EMB):
                W[q][j] -= step * gW[q][j] / n
        if learn:
            for i, g in gT.items():
                for j in range(EMB):
                    T[i][j] -= step * g[j] / n
    return T, W, b


def accuracy(P, D):
    T, W, b = P
    return sum(max(range(4), key=lambda q: sum(
        W[q][j] * sum(T[i][j] for i in z) / len(z) for j in range(EMB)) + b[q]) == y
        for z, y in D) / len(D)


SIZES = (10, 20, 40, 80, 160)
SETUPS = (("from scratch", False, True), ("random table, frozen", False, False),
          ("pretrained, frozen", True, False), ("pretrained, fine-tuned", True, True))
RESULTS = {}
print(f"{'setup':<25}{'learned':>9}" + "".join(f"{'n=' + str(n):>9}" for n in SIZES)
      + f"{'runs':>8}")
for label, pretrained, learn in SETUPS:
    RESULTS[label] = {}
    for n in SIZES:
        RESULTS[label][n] = [accuracy(train(T0 if pretrained else make_table(SEED + 900 + k),
                                            LABELED_ENC[:n], SEED + 30 + k, learn), HELD_ENC)
                             for k in (1, 2)]
    print(f"{label:<25}{4 * EMB + 4 + (V * EMB if learn else 0):>9}"
          + "".join(f"{sum(RESULTS[label][n]) / 2:>9.4f}" for n in SIZES)
          + f"{max(abs(v[0] - v[1]) for v in RESULTS[label].values()):>8.4f}")
print(f"domain baseline {BASELINE:.4f} (last column: the largest difference of the two runs)")
for label, n in (("pretrained, frozen", 10), ("pretrained, fine-tuned", 20)):
    h = sum(RESULTS[label][n]) / 2
    y = [m for m in SIZES if sum(RESULTS["from scratch"][m]) / 2 >= h]
    print(f"{label} n={n} -> {h:.4f}; from scratch reaches the same number with "
          f"{y[0] if y else '>160'} examples")

setup                      learned     n=10     n=20     n=40     n=80    n=160    runs
from scratch                   682   0.5050   0.5850   0.7770   0.8740   0.9160  0.0460
random table, frozen            28   0.4510   0.4630   0.5000   0.5050   0.5000  0.1320
pretrained, frozen              28   0.6050   0.6610   0.7470   0.7470   0.7550  0.0020
pretrained, fine-tuned         682   0.5880   0.7480   0.8180   0.8780   0.9070  0.0080
domain baseline 0.4420 (last column: the largest difference of the two runs)
pretrained, frozen n=10 -> 0.6050; from scratch reaches the same number with 40 examples
pretrained, fine-tuned n=20 -> 0.7480; from scratch reaches the same number with 40 examples

The second row is the most important row of this table, and it has to be written before reading. When a randomly initialized table that has learned nothing is frozen, it stays below 0.51 in five out of five measurements — about 0.06 above the baseline. Averaging itself, without learning which vector the words sit on, gives almost nothing. The 0.7550 of the third row is therefore the counterpart of what masked prediction learned: the same 28 parameters, the same averaging, the only difference being that the table came from unlabeled text, and the difference is 0.2550.

This is the reverse of the result of the Introduction to Machine Learning course, fundamental-concepts/03. There the learned part of the proxy task had added minus 0.0027 to the real task, and the whole of the contribution had come from the carried-over raw observation. Here there is no carried-over raw observation — what is carried over is only the learned table, and the contribution is 0.2550. The difference between the two lessons lies in how much the proxy task is related: the link between predicting the reading of a period and suspected illicit use is weak, while the link between predicting the hidden word and the event kind is direct, because both come from the same word co-occurrences.

The label count columns give the second answer. With twenty labels the fine-tuned setup gives 0.7480 and training from scratch 0.5850: the difference is 0.1630, and since it is larger than three times the run variability of 0.0460, it counts as measured. Training from scratch reaches the same number only with forty labels. At one hundred sixty, from scratch gives 0.9160 and fine-tuned 0.9070 — the difference has turned to minus 0.0090, that is, it has closed. What pretraining gained is not the final number but the twenty labels that did not have to be collected to reach it.

The last column stands in the table separately. The two runs of the random frozen setup differ by 0.1320: every number printed in that row is an accident that depends on which random table happened to fall. The same body with the pretrained table differs by 0.0020. Pretraining does not only raise the average, it also lowers the scatter between runs; this second thing is never visible in a table that prints a single number.

Summary

• The proxy task is masked prediction and it spends no labels: on its own measure it gives 0.3450 where the baseline is 0.0419, a difference of 0.3032.
• The domain baseline is 0.4420; with twenty labels training from scratch gives 0.5850, pretraining and fine-tuning give 0.7480, and training from scratch reaches this number only with forty labels.
• The difference closes with the number of labels: at one hundred sixty labels the fine-tuned setup falls 0.0090 below training from scratch.
• The control setup separates the source of the difference: when a random table that has never been trained is frozen, it stays below 0.51 in all five measurements; the pretrained table gives 0.7550 with the same 28 parameters.
• Pretraining also lowers the scatter between runs: the two runs of the random table differ by 0.1320, the two runs of the pretrained table by 0.0020.

Next Step

In six lessons the output of the model was always a label: a class number, or a class number per token. The correct answer was at hand and the measure was a single comparison. When the output of a model is the text itself, this measure cannot be built: there is no single correct sequence for the summary of a note, and other sequences carrying the same content are also correct. In this case the criterion turns into the overlap between the candidate text and the reference text. The next lesson counts what this criterion rewards and shows this: a procedure that gives the input back as it is, one that has learned nothing, gets a high number on an overlap-based criterion.