Adaptation Methods

Fine-tuning carries a model into a new domain, and it does this by moving all the weights. The previous lesson wrote the cost of this as example count and epoch count. There is one more cost that was not written: for every domain adapted, a full copy of the model is stored. Eight domains means eight copies.

This lesson tries moving less. Three setups are placed side by side on the same adaptation task, full update, layer freezing and a low-rank update, and the updated-parameter share and the accuracy of each are printed together. The number of stored numbers is written next to them, because the real cost of adaptation is paid not during training but after it.

• LA31. The code is a simulator. The local model, vocabulary, records and labels are fictional; no real endpoint is called. Seed 20260218.
• LA32. The model is three layers: an input lookup table, a body matrix and a head. All of the weights are in our hands; this is the distinguishing side of a model you host yourself.
• LA33. The two domains are separate: the model is pretrained on general-spelling fictional tickets, then adapted to the field team’s abbreviated fictional records. A share of the records get a misleading suffix carrying the abbreviation of another topic.
• LA34. The low-rank update does not change the weight matrix in place; next to it, it adds the product of two thin matrices. Because the second starts from zero, the base model is not disturbed at first. The singular value decomposition lesson in the Unsupervised Learning course had measured that a matrix can be written approximately as a low-rank product; the structure here is the update side of the same idea.
• LA35. The head is updated in every setup; what separates the setups is how much of the body and the lookup table moves. The two runs are taken with the pretraining seed (20260218 and 20260219), the sets do not change.

Layers and the Weight Budget

On a distant endpoint the only thing measurable was the surface. On a model you host yourself, what can be measured is the numbers themselves: how many there are, which is in which layer, how many are saved.

# SIMULATOR -- FICTIONAL. The local model, vocabulary, records and labels are written
# inside the lesson; no real model endpoint is called. Seed 20260218.
from math import exp

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, a piece is at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"]
GENERAL = [("bu ayki fatura tutari gecen aya gore yuksek geldi", "invoice"),
           ("faturami odedim ama sistemde borc gorunuyor", "invoice"),
           ("evde su akmiyor bir ariza var galiba", "breakdown"),
           ("baglanti bozuk oldu ve hic calismiyor", "breakdown"),
           ("sayac okumasi yanlis girilmis endeks tutmuyor", "reading"),
           ("sayac degisimi icin randevu istiyorum", "reading"),
           ("hangi tarife basamagindayim ogrenmek istiyorum", "tariff"),
           ("birim ucret abone tipine gore degisiyor mu", "tariff"),
           ("mahallede kesinti var ne zaman gelecek", "outage"),
           ("planli kesinti duyurusu nereden takip edilir", "outage")]
FIELD = [("fat sapma itiraz dosyasi acildi", "invoice"),
         ("tahak farki cikti fat duzelt", "invoice"),
         ("fat kalem hatasi bildirildi", "invoice"),
         ("fat iade sureci baslatildi", "invoice"),
         ("arz hat kesik ekip yonlendirildi", "breakdown"),
         ("pano arz kodu iki bir dort", "breakdown"),
         ("arz tekrar etti kalici cozum", "breakdown"),
         ("arz gecici onarim yapildi", "breakdown"),
         ("syc endeks sapmasi tespit", "reading"),
         ("syc degisim isi planlandi", "reading"),
         ("syc muhur kontrolu yapildi", "reading"),
         ("syc seri numarasi eslesmedi", "reading"),
         ("trf basamak gecisi talebi", "tariff"),
         ("trf birim degisimi islendi", "tariff"),
         ("trf sozlesme guncellemesi bekliyor", "tariff"),
         ("trf indirim kosulu incelendi", "tariff"),
         ("kes duyurusu bolgeye islendi", "outage"),
         ("kes suresi uzadi bilgilendirme", "outage"),
         ("kes sonrasi enerji verildi", "outage"),
         ("kes plani ertelendi yeni saat", "outage")]
MISLEADING = [" fat kaydi da acildi", " arz notu eklendi",
              " syc formu istendi", " trf grubu kontrol edildi"]


def make_set(pool, n, seed, decoy_share=0.0):
    r, k = rng(seed), []
    for i in range(n):
        c = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
        candidates = [x for x in pool if x[1] == c]
        m = candidates[int(r() * len(candidates))][0]
        if r() < decoy_share:           # a suffix carrying the abbreviation of another topic
            m += MISLEADING[int(r() * 4)]
        k.append((m, c))
    return k


PRETRAIN = make_set(GENERAL, 150, SEED)                 # pretraining domain
ADAPT = make_set(FIELD, 48, SEED + 7, 0.30)             # field spelling, adaptation examples
TEST = make_set(FIELD, 72, SEED + 11, 0.30)             # field spelling, test
VOCAB = {}
for m in [x[0] for x in GENERAL + FIELD] + MISLEADING:
    for t in tokenize(m):
        VOCAB.setdefault(t, len(VOCAB))
V, H, C = len(VOCAB), 12, len(TOPICS)


def encode(k):
    return [(sorted({VOCAB[t] for t in tokenize(m)}), TOPICS.index(c)) for m, c in k]


PRE_E, ADAPT_E, TEST_E = encode(PRETRAIN), encode(ADAPT), encode(TEST)


def build(seed):
    """Input lookup table, body matrix, head. The body starts around the identity matrix."""
    r = rng(seed)
    return [[[(r() - 0.5) * 0.4 for _ in range(H)] for _ in range(V)],
            [[1.0 * (k == j) + (r() - 0.5) * 0.2 for j in range(H)] for k in range(H)],
            [[(r() - 0.5) * 0.4 for _ in range(C)] for _ in range(H)]]


def multiply(inputs, W, n):
    """inputs: (row, value) pairs; sparse multiplication."""
    h = [0.0] * n
    for i, v in inputs:
        Wi = W[i]
        for j in range(n):
            h[j] += v * Wi[j]
    return h


def forward(p, tok, addon=None):
    """If addon is given, a low-rank update is added to every layer (second section)."""
    layer, mid = [(t, 1.0) for t in tok], []
    for n in (0, 1):
        h, z = multiply(layer, p[n], H), None
        if addon:
            U, Vt = addon[n]
            z = multiply(layer, U, len(Vt))
            for j, x in enumerate(multiply(list(enumerate(z)), Vt, H)):
                h[j] += x
        mid.append((layer, h, z))
        layer = [(j, x) for j, x in enumerate(h) if x > 0.0]    # rectification, sparse
    return mid, layer, multiply(layer, p[2], C)


def backward(p, tok, y, addon, step, which):
    """which: full | head | add_on. The head is updated in every setup, the others depend on the choice."""
    mid, layer, o = forward(p, tok, addon)
    mx = max(o)
    e = [exp(x - mx) for x in o]
    s = sum(e)
    d = [x / s for x in e]
    d[y] -= 1.0
    B, da = p[2], [0.0] * H
    for k, v in layer:
        da[k] = sum(d[j] * B[k][j] for j in range(C))
        for j in range(C):
            B[k][j] -= step * d[j] * v
    for n in (1, 0):
        inputs, h, z = mid[n]
        W, U, Vt = p[n], *(addon[n] if addon else (None, None))
        dh = [da[j] if h[j] > 0.0 else 0.0 for j in range(H)]
        if n == 1:
            da = [sum(dh[j] * W[k][j] for j in range(H)) for k in range(H)]
            if addon:
                dz = [sum(dh[j] * Vt[q][j] for j in range(H)) for q in range(len(Vt))]
                for k in range(H):
                    da[k] += sum(U[k][q] * dz[q] for q in range(len(Vt)))
        if which == "full":
            for i, v in inputs:
                for j in range(H):
                    W[i][j] -= step * dh[j] * v
        elif which == "add_on":
            dz = [sum(dh[j] * Vt[q][j] for j in range(H)) for q in range(len(Vt))]
            for q in range(len(Vt)):
                for j in range(H):
                    Vt[q][j] -= step * dh[j] * z[q]
            for i, v in inputs:
                for q in range(len(Vt)):
                    U[i][q] -= step * dz[q] * v


def accuracy(p, k, addon=None):
    return sum(max(range(C), key=lambda j: forward(p, tok, addon)[2][j]) == y
               for tok, y in k) / len(k)


def pretrain(seed):
    p = build(seed)
    for e in range(30):
        for tok, y in PRE_E:
            backward(p, tok, y, None, 0.10, "full")
    return p


BASE_A, BASE_B = pretrain(SEED), pretrain(SEED + 1)
print(f"vocabulary {V} pieces , hidden width {H} , classes {C}")
print(f"input {V * H} + body {H * H} + head {H * C} = {V * H + H * H + H * C} numbers")
print("run   pretraining domain   field domain")
for name, p in (("A", BASE_A), ("B", BASE_B)):
    print(f"  {name}   {accuracy(p, PRE_E):>18.4f}   {accuracy(p, TEST_E):>12.4f}")

vocabulary 174 pieces , hidden width 12 , classes 5
input 2088 + body 144 + head 60 = 2292 numbers
run   pretraining domain   field domain
  A               1.0000         0.2500
  B               1.0000         0.0833

On the pretraining domain accuracy is 1.0000 in both runs; the model has learned its own domain. When it moves to field spelling, the same model gives 0.2500 and 0.0833. In a five-class task random choice is around 0.2000, so the base model knows nothing on field records; the 0.1667 gap between the two runs is the proof of this, what is measured is not knowledge but noise. Adaptation is unavoidable and the question is which numbers will move.

Updated-Parameter Share and Accuracy

The setups are run with the same forty-eight adaptation records, the same twenty-five loops and the same step size; the only thing that changes is which share backpropagation touches.

from copy import deepcopy


def build_addon(rank, seed):
    """Each low-rank update is a pair of U (rows x rank) and Vt (rank x H). Vt starts from zero,
    so at the first moment of adaptation the update adds nothing and the base model is not disturbed."""
    r = rng(seed)
    return [([[(r() - 0.5) * 0.3 for _ in range(rank)] for _ in range(n)],
             [[0.0] * H for _ in range(rank)]) for n in (V, H)]


def adapt(base, which, rank, seed, loops=25, step=0.08):
    p = deepcopy(base)
    addon = build_addon(rank, seed + 3) if rank else None
    for e in range(loops):
        for tok, y in ADAPT_E:
            backward(p, tok, y, addon, step, which)
    return accuracy(p, TEST_E, addon), accuracy(p, PRE_E, addon)


TOTAL = V * H + H * H + H * C
SETUPS = [("full update", "full", 0), ("layer freezing", "head", 0)] + \
         [(f"low-rank update r={r}", "add_on", r) for r in (1, 2, 3, 4)]
print("setup                updated   share  field A  field B  general A  general B")
print(f"{'no adaptation':<20} {0:>7}  {0.0:.4f}  {accuracy(BASE_A, TEST_E):>7.4f}"
      f"  {accuracy(BASE_B, TEST_E):>7.4f}  {accuracy(BASE_A, PRE_E):>9.4f}"
      f"  {accuracy(BASE_B, PRE_E):>9.4f}")
for name, which, rank in SETUPS:
    n = TOTAL if which == "full" else (H * C if which == "head"
                                       else rank * (V + H) + rank * 2 * H + H * C)
    a, b = adapt(BASE_A, which, rank, SEED), adapt(BASE_B, which, rank, SEED + 1)
    print(f"{name:<20} {n:>7}  {n / TOTAL:.4f}  {a[0]:>7.4f}  {b[0]:>7.4f}"
          f"  {a[1]:>9.4f}  {b[1]:>9.4f}")

print("\nadaptation to eight separate domains, number of stored numbers:")
for name, n in (("full update", TOTAL), ("layer freezing", H * C),
                ("low-rank update r=2", 2 * (V + H) + 2 * 2 * H + H * C)):
    shared = 0 if n == TOTAL else TOTAL
    print(f"  {name:<20} {shared} + 8 x {n} = {shared + 8 * n}")

print("\nas the hidden width grows (independent of the run, count only):")
print("  H        full  freezing  r=2 update  update share")
for h in (H, 64, 256):
    full = V * h + h * h + h * C
    addon2 = 2 * (V + h) + 2 * 2 * h + h * C
    print(f"  {h:<5} {full:>7}  {h * C:>8}  {addon2:>10}  {addon2 / full:>12.4f}")

setup                updated   share  field A  field B  general A  general B
no adaptation              0  0.0000   0.2500   0.0833     1.0000     1.0000
full update             2292  1.0000   1.0000   0.9861     1.0000     1.0000
layer freezing            60  0.0262   0.6944   0.7500     0.5133     0.1733
low-rank update r=1      270  0.1178   0.7639   0.7222     0.5133     0.2733
low-rank update r=2      480  0.2094   0.9583   0.9306     1.0000     0.5533
low-rank update r=3      690  0.3010   1.0000   0.9722     1.0000     1.0000
low-rank update r=4      900  0.3927   0.9861   1.0000     1.0000     1.0000

adaptation to eight separate domains, number of stored numbers:
  full update          0 + 8 x 2292 = 18336
  layer freezing       2292 + 8 x 60 = 2772
  low-rank update r=2  2292 + 8 x 480 = 6132

as the hidden width grows (independent of the run, count only):
  H        full  freezing  r=2 update  update share
  12       2292        60         480        0.2094
  64      15552       320        1052        0.0676
  256    111360      1280        3164        0.0284

Full update moves all 2292 numbers and gives 1.0000 / 0.9861. Layer freezing works with 0.0262 of that, that is, sixty numbers, and stays at 0.6944 / 0.7500: forty-eight examples are enough to teach the head a new mapping, but because the body still does not recognize the field abbreviations, accuracy stays half-way. The low-rank update at rank two climbs to 0.9583 / 0.9306 with a share of 0.2094; its distance from full update is between 0.0417 and 0.0555, and its distance from layer freezing is between 0.1806 and 0.2639. Most of the gap is closed with two tenths of the updated share.

The rank ladder says where to stop. Rank one is not enough (0.7639 / 0.7222): a single direction cannot carry the shift between the two spellings. Rank three gives 1.0000 / 0.9722 and rank four 0.9861 / 1.0000; the means are 0.9861 and 0.9931, so the mean gain of raising the rank from three to four is 0.0070. The two-run gaps of the same rows are 0.0278 and 0.0139. The gain is below the gap, and in return 210 more numbers are stored per adaptation: the cost of the added structure is certain, its gain is unmeasurable.

The last two columns show the unexpected. Layer freezing, although it protects the body, drops accuracy on the previous domain to 0.5133 / 0.1733; the protected body does not help, because the head the two domains share was rewritten. Full update, on the other hand, keeps the previous domain at 1.0000: in the lookup table only the rows that appear in field records moved, and the rows of general spelling were not touched. Freezing a layer does not mean preserving that layer’s ability; what is preserved is the weights, what is lost is the behavior.

Stored Count and Scale

The row of adaptation to eight domains shows where the decision is paid for. In full update the base cannot be shared; eight copies come to 18336 numbers. In layer freezing the base is stored once and each domain adds only its own head: 2772. In the low-rank update the base is again stored once and each domain adds 480 numbers: 6132, that is, a third of full update.

The scale table says this ratio is not constant. When the rank is held fixed, the cost of the low-rank update grows linearly with the hidden width, and the cost of full update grows quadratically in the body. At width 12 the share of the low-rank update is 0.2094, at 64 0.0676, at 256 0.0284. This count is independent of the run, and it explains why the discussion of adaptation methods starts with large models.

Summary

• The fictional local model is 2292 numbers (input 2088, body 144, head 60), and its unadapted state stays at 0.2500 and 0.0833 on field spelling, that is, at the level of random choice.
• Full update moves all of it and gives 1.0000 / 0.9861; its cost is a full copy for each domain.
• Layer freezing stays at 0.6944 / 0.7500 with a 0.0262 share and drops accuracy on the previous domain to 0.5133 / 0.1733: the frozen weights are preserved, the behavior is not.
• The low-rank update gives 0.9583 / 0.9306 with a 0.2094 share; its distance from full update is between 0.0417 and 0.0555.
• The mean gain of raising the rank from three to four is 0.0070, the two-run gaps of the same rows are 0.0278 and 0.0139: the gain counts as unmeasured, the 210 extra numbers are certain.
• The stored count over eight domains is 6132 against 18336, and when the hidden width goes from 12 to 256 the share of the low-rank update falls from 0.2094 to 0.0284.

Next Step

In this lesson the weights of the large model were in our hands. Weights are not always in hand; most of the time the only thing in hand is the output of a large model. That output can be used like a label set and made the input for training a small model. The next lesson measures this: what does a small model trained with the labels a large model produced gain compared with a model of the same size trained only with human labels, how much of the teacher’s accuracy does it capture, and in which task does the gap not close.