Transfer Learning

In the previous lesson the embedding table was learned with the labels of the task itself, and two hundred examples were enough for it. Where the label is expensive, this assumption fails. A subscriber record marked as suspect is the equivalent of an hour an inspector spent in the field; there may be twenty or forty of them at hand. Learning all hundred and three parameters from twenty labels is a case where capacity exceeds the data, and generalization of the Introduction to Machine Learning course had measured this.

There is one more way. The same codes also answer another question, and for that question labels are plentiful. The representation can be learned there and carried over here. The question of this lesson is twofold: which number does carrying it move, and with how many labels. The second question matters more than the first, because the value of transfer lies not in the final number but in how many labels it takes to reach that number.

• AR73. The sequence setup of the Architectures topic was built in architectures/06 and the event codes in architectures/08; they are fictional. In this lesson a second fictional label is added. Seed 20260218.
• AR74. The source task is fictional and its label is plentiful: the presence of a maintenance code in at least six of the twelve steps. There are ten maintenance codes.
• AR75. The target task is the task of architectures/08: a suspect code in at least five steps. There are eight suspect codes and all eight are among the maintenance codes; the maintenance set has two more codes besides them. The two tasks are related but not the same.
• AR76. The pretrained model is produced in this lesson. No ready-made model, weight repository or external source is used; the source task is trained here with three hundred examples and is a model.
• AR77. The source and the target work on separate sequences: the source is the first three hundred sequences, the target pool is the next two hundred forty, and the held-out set is the last one hundred eighty. The pretrained model never sees a single label of the target task.
• AR78. Three setups are compared: from scratch (the table is randomly initialized), transfer + freezing (the table is taken from the source and never updated), transfer + fine-tuning (the table is taken from the source and updated).
• AR79. The number of target labels is measured at four values: 20, 40, 80, 160. The held-out set is the same in all four.
• AR80. The baseline is the most frequent class and is read from the held-out set; the baseline does not change when the label count changes.
• AR81. Every number is printed with two runs; the runs differ in the initialization seed.

The Source Task and the Pretrained Model

The subject of transfer is not a file but a representation. What is carried here is the embedding table: the three-dimensional vectors of the twenty-four codes. The source task places this table according to its own question, and the target task inherits it.

# common.py — MODEL. The FICTIONAL sequence set of `architectures/06` + the FICTIONAL event codes of `08`.
import math

SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12
CODES = 24
MAINT = (1, 2, 4, 5, 7, 9, 13, 16, 20, 22)     # FICTIONAL source task: maintenance codes
SUSPECT = (1, 4, 5, 9, 13, 16, 20, 22)         # FICTIONAL target task: suspect codes


def generator(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_sequences(n, seed):
    D = []
    for i in range(n):
        r = generator(seed + i)
        r()
        [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)]
        r()
        o = generator(seed + 8000 + i)
        codes = [int(o() * CODES) for _ in range(STEPS)]
        D.append({"codes": codes, "source": 1 if sum(c in MAINT for c in codes) >= 6 else 0,
                  "y": 1 if sum(c in SUSPECT for c in codes) >= 5 else 0})
    return D


DATA = build_sequences(720, SEED)
SOURCE, POOL, TEST = DATA[:300], DATA[300:540], DATA[540:]
rate = sum(x["y"] for x in TEST) / len(TEST)
BASELINE = max(rate, 1 - rate)
SHARED = [c for c in SUSPECT if c in MAINT]
print(f"codes {CODES}, maintenance {len(MAINT)}, suspect {len(SUSPECT)}, shared {len(SHARED)}")
print(f"source task rate {sum(x['source'] for x in SOURCE) / len(SOURCE):.4f}, "
      f"source examples {len(SOURCE)}")
print(f"target task rate held-out {rate:.4f}, baseline {BASELINE:.4f}, "
      f"pool {len(POOL)}")

codes 24, maintenance 10, suspect 8, shared 8
source task rate 0.3533, source examples 300
target task rate held-out 0.3556, baseline 0.6444, pool 240

The relation between the two tasks is in a single line: all eight suspect codes are in the maintenance set, but the maintenance set has two extra codes. A representation that solves the source task carries the separation the target task wants approximately; it does not carry it exactly.

HIDDEN, DIM = 6, 3


def represent(T, codes):
    h = [0.0] * DIM
    for c in codes:
        for j in range(DIM):
            h[j] += T[c][j] / STEPS
    return h


def new_table(seed):
    r = generator(seed)
    return [[(r() + r() + r() - 1.5) * 1.1 for _ in range(DIM)] for _ in range(CODES)]


def train_model(seed, T, D, field, epochs, learn, lr=0.9):
    r = generator(seed + 500)
    g = lambda: (r() + r() + r() - 1.5) * 0.9
    W1 = [[g() for _ in range(DIM)] for _ in range(HIDDEN)]
    b1, w, b, n = [0.0] * HIDDEN, [g() for _ in range(HIDDEN)], [0.0], len(D)
    for _ in range(epochs):
        gW1 = [[0.0] * DIM for _ in range(HIDDEN)]
        gb1, gw, gb = [0.0] * HIDDEN, [0.0] * HIDDEN, 0.0
        gT = [[0.0] * DIM for _ in range(CODES)] if learn else None
        for x in D:
            h = represent(T, x["codes"])
            z = [sum(W1[i][j] * h[j] for j in range(DIM)) + b1[i] for i in range(HIDDEN)]
            o = sum(w[i] * z[i] for i in range(HIDDEN) if z[i] > 0) + b[0]
            df = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - x[field]
            gb += df
            da = [df * w[i] if z[i] > 0 else 0.0 for i in range(HIDDEN)]
            for i in range(HIDDEN):
                gw[i] += df * max(0.0, z[i])
                gb1[i] += da[i]
                for j in range(DIM):
                    gW1[i][j] += da[i] * h[j]
            if learn:                   # with freezing the gradient of the table is never computed
                dh = [sum(da[i] * W1[i][j] for i in range(HIDDEN)) for j in range(DIM)]
                for c in x["codes"]:
                    for j in range(DIM):
                        gT[c][j] += dh[j] / STEPS
        for i in range(HIDDEN):
            w[i] -= lr * gw[i] / n
            b1[i] -= lr * gb1[i] / n
            for j in range(DIM):
                W1[i][j] -= lr * gW1[i][j] / n
        b[0] -= lr * gb / n
        if learn:
            for c in range(CODES):
                for j in range(DIM):
                    T[c][j] -= lr * gT[c][j] / n
    return {"T": T, "W1": W1, "b1": b1, "w": w, "b": b[0]}


def accuracy(P, D, field="y"):
    s = 0
    for x in D:
        h = represent(P["T"], x["codes"])
        z = [sum(P["W1"][i][j] * h[j] for j in range(DIM)) + P["b1"][i]
             for i in range(HIDDEN)]
        o = sum(P["w"][i] * max(0.0, z[i]) for i in range(HIDDEN)) + P["b"]
        s += (o > 0) == (x[field] == 1)
    return s / len(D)


def copy_table(T):
    return [u[:] for u in T]


PRETRAINED = []
print(f"source task examples {len(SOURCE)}, epochs 400, embedding dimension {DIM}, "
      f"hidden units {HIDDEN}")
for k in (1, 2):
    P = train_model(SEED + k, new_table(SEED + 900 + k), SOURCE, "source", 400, True)
    PRETRAINED.append(P)
    print(f"pretrained model run {k}: training on the source task "
          f"{accuracy(P, SOURCE, 'source'):.4f}, held-out "
          f"{accuracy(P, TEST, 'source'):.4f}")
print(f"these two models never saw any label of the target task")

source task examples 300, epochs 400, embedding dimension 3, hidden units 6
pretrained model run 1: training on the source task 1.0000, held-out 1.0000
pretrained model run 2: training on the source task 1.0000, held-out 1.0000
these two models never saw any label of the target task

Freezing, Fine-Tuning and How Many Labels

Transfer has two forms. In freezing the carried table is fixed and only the body on top of it is learned: thirty-one parameters. In fine-tuning the table is updated as well: one hundred three parameters. The choice between the two depends on how many labels are at hand, and this dependence can itself be measured.

SIZES, SETUPS = (20, 40, 80, 160), (("from scratch", False, True),
                                    ("transfer + freezing", True, False),
                                    ("transfer + fine-tuning", True, True))
RESULT = {}
print(f"target epochs 250, baseline {BASELINE:.4f}, held-out {len(TEST)}")
print(f"{'setup':<24}{'learned':>10}{'run':>7}"
      + "".join(f"{'n=' + str(n):>9}" for n in SIZES))
for name, from_source, learn in SETUPS:
    RESULT[name] = {}
    for k in (1, 2):
        row = []
        for n in SIZES:
            T = copy_table(PRETRAINED[k - 1]["T"]) if from_source else new_table(SEED + 900 + k)
            P = train_model(SEED + 30 + k, T, POOL[:n], "y", 250, learn)
            RESULT[name].setdefault(n, []).append(accuracy(P, TEST))
            row.append(RESULT[name][n][-1])
            if n == SIZES[-1] and k == 1:
                RESULT[name]["T"] = P["T"]
        print(f"{name if k == 1 else '':<24}"
              f"{HIDDEN * DIM + 2 * HIDDEN + 1 + (0 if not learn else CODES * DIM):>10}"
              f"{k:>7}" + "".join(f"{v:>9.4f}" for v in row))

target epochs 250, baseline 0.6444, held-out 180
setup                      learned    run     n=20     n=40     n=80    n=160
from scratch                   103      1   0.7222   0.8056   0.8167   0.9556
                               103      2   0.7167   0.8389   0.8611   0.9722
transfer + freezing             31      1   0.8333   0.8278   0.8389   0.8333
                                31      2   0.8500   0.8444   0.8278   0.8278
transfer + fine-tuning         103      1   0.8111   0.8556   0.8556   0.9556
                               103      2   0.8056   0.8500   0.8556   0.9667

The freezing row is a horizontal line: between 0.8278 and 0.8500, it hardly moves with the label count. Thirty-one parameters can be learned from twenty labels, but one hundred sixty labels do not find a new place to go in the same thirty-one parameters. The ceiling of transfer is in the carried table.

def mean_acc(name, n):
    return sum(RESULT[name][n]) / 2


print(f"{'examples':>9}{'from scratch':>14}{'freezing':>10}{'fine-tuning':>13}"
      f"{'freezing difference':>21}{'fine-tuning difference':>24}")
for n in SIZES:
    a, b, c = (mean_acc(k, n) for k, *_ in SETUPS)
    print(f"{n:>9}{a:>14.4f}{b:>10.4f}{c:>13.4f}{b - a:>+21.4f}{c - a:>+24.4f}")
target = mean_acc("transfer + freezing", 20)
enough = [n for n in SIZES if mean_acc("from scratch", n) >= target]
print(f"freezing with 20 examples {target:.4f}; from scratch reaches the same "
      f"figure with {enough[0] if enough else '>160'} examples")


def pairwise(T, P, Q):
    d = [sum((T[i][j] - T[k][j]) ** 2 for j in range(DIM)) ** 0.5
         for i in P for k in Q if i != k]
    return sum(d) / len(d)


def separation(T, G):
    A = [c for c in range(CODES) if c in G]
    B = [c for c in range(CODES) if c not in G]
    return pairwise(T, A, B) / ((pairwise(T, A, A) + pairwise(T, B, B)) / 2)


print(f"{'table':<28}{'maintenance separation':>24}{'suspect separation':>20}")
for name, T in (("from the source task", PRETRAINED[0]["T"]),
                ("after fine-tuning, n=160", RESULT["transfer + fine-tuning"]["T"]),
                ("from scratch, n=160", RESULT["from scratch"]["T"])):
    print(f"{name:<28}{separation(T, MAINT):>24.4f}{separation(T, SUSPECT):>20.4f}")

 examples  from scratch  freezing  fine-tuning  freezing difference  fine-tuning difference
       20        0.7194    0.8417       0.8083              +0.1222                 +0.0889
       40        0.8222    0.8361       0.8528              +0.0139                 +0.0306
       80        0.8389    0.8333       0.8556              -0.0056                 +0.0167
      160        0.9639    0.8306       0.9611              -0.1333                 -0.0028
freezing with 20 examples 0.8417; from scratch reaches the same figure with 160 examples
table                         maintenance separation  suspect separation
from the source task                          3.6373              2.4767
after fine-tuning, n=160                      2.4685              3.7991
from scratch, n=160                           2.1087              3.3228

The first table reads the difference that transfer adds together with the number of examples and does not give a single number. With twenty labels freezing stands 0.1222 above training from scratch; with forty the difference drops to 0.0139; with eighty the sign turns, and with one hundred sixty freezing falls below training from scratch by 0.1333. Fine-tuning is above at three of the four points, but at one hundred sixty its difference falls to 0.0028, that is, within the measurement noise.

The second line is the real number of this lesson. Freezing gives 0.8417 with twenty labels; training from scratch reaches the same figure only with one hundred sixty labels — eight times as many. What transfer buys is not the final number but the labels that are not collected to reach it. If one field visit is assumed per label, the unit of the measure is not accuracy but one hundred forty visits.

The minus sign in the one hundred sixty row deserves a pause. There, transfer with freezing is beaten by the setup that transfers nothing; the carried representation has stopped being a starting point and has become a constraint. Transfer does not always pay off, and whether it pays off is known only when the setup without transfer is run on the same held-out set. The baseline of this measure is not the most frequent class but training from scratch itself.

Run-to-run variability determines the reading of this lesson in two places. With twenty labels the difference between the runs of the three setups is at most 0.0167, while what transfer adds is 0.1222 — that is, the difference is seven times the run-to-run variability and counts as measured. With one hundred sixty, however, the difference of fine-tuning from training from scratch is 0.0028, and the two runs of the same setup differ by 0.0111; there the difference is lost inside the run, and saying “fine-tuning is a little better” is an unmeasured sentence.

The third table says why transfer stops somewhere. The table from the source carries a maintenance separation of 3.6373 but a suspect separation of only 2.4767; the two extra maintenance codes have settled next to the suspects in the table, and a frozen table cannot correct this. Fine-tuning swaps the two numbers: the maintenance separation drops to 2.4685 while the suspect separation rises to 3.7991. The table trained from scratch also brings the suspect separation to 3.3228, but it spends eight times the labels to do so.

Summary

• The pretrained model was produced in this lesson from a fictional source task; on the source task both runs give 1.0000 and the model never saw a single label of the target task.
• With twenty labels the baseline is 0.6444, training from scratch 0.7194, freezing 0.8417 and fine-tuning 0.8083; at this point transfer adds a difference of up to 0.1222.
• Freezing learns thirty-one parameters and does not move with the label count: it stays flat at around 0.83 in all four measurements and at one hundred sixty labels falls 0.1333 below training from scratch.
• The measure of transfer is not accuracy but the number of labels: training from scratch reaches the 0.8417 that freezing gives with twenty labels only with one hundred sixty labels.
• The cause of the ceiling is the carried representation: the source table carries a maintenance separation of 3.6373 and a suspect separation of 2.4767; after fine-tuning these two numbers become 2.4685 and 3.7991.

Next Step

In nine lessons every architecture predicted a label. The input was given, the output was reduced to a single number, and the measure was always of the same form: how many correct on the held-out set. A network can have another job besides predicting a label — reproducing the input itself, or building new examples that resemble the input. What is the measure of such a network, because there is no label to compare against? The next lesson takes up two generative arrangements that work over a latent space: the autoencoder, measured by the reconstruction error (with reference to the measure of the Unsupervised Learning course), and the adversarial arrangement in which two networks are trained against each other; in the latter, what is measured is not performance but how far apart two runs of the same arrangement end up.