Generative Models

In nine lessons every architecture predicted a label, and the measure was always of the same form: how many correct on the held-out set. The job of a network may not be to predict a label. Reconstructing the input itself from a few numbers, or producing new sequences that resemble the input, is also a task, and in neither is there a label to compare against.

The question of this lesson is about the measure: which number is written when there is no label. Two arrangements are taken up. The first reconstructs the input from a latent space, and its error can be measured directly. In the second two networks are trained against each other and there is not even a single error; what is measured there is not performance but how far apart two runs of the same arrangement end up.

• AR82. The sequence setup of the Architectures topic was built in architectures/06 and is fictional. In this lesson the sequence is used in its raw form: subscriber level, periodic component and the residual of the earlier lessons. The earlier lessons took the sequence scaled as a deviation; the compressible structure lies exactly in the components extracted there.
• AR83. The columns are scaled from the training set only and the error is in units of standard deviation. This is the same unit as dimensionality-reduction in the Unsupervised Learning course.
• AR84. The baseline is reconstruction with the mean sequence, which is the case where the latent dimension is zero. This is the plainest reconstruction that does not learn.
• AR85. The reconstruction error procedure was built in the Unsupervised Learning course and is not repeated here. There the directions were computed from the covariance matrix, and against the 3.3411 of reconstructing with the column means, two components had given 2.7762. Here the directions are not computed; they are learned with gradient descent.
• AR86. The autoencoder is a linear encoder and a linear decoder; because the data is centered, no bias term is written. It is written inside the lesson and is a model.
• AR87. The adversarial generative network is an arrangement built from two networks: the generator builds a sequence from a latent vector, the discriminator reduces a sequence to a single number. The two are updated in turn.
• AR88. The measure of the adversarial arrangement is not accuracy. The step means and spreads of the generated sequences are compared with those of the real set, and how far apart the two runs end up is printed.
• AR89. This lesson takes up only the structure and the measure. No production application, forgery or detection-evasion procedure is written; these topics are left to the defensive framework of the Cybersecurity and Application Security and DevSecOps curricula.
• AR90. Every number is printed with two runs; the runs differ in the initialization seed.

Autoencoder and Reconstruction Error

The autoencoder is two parts. The encoder reduces the twelve-number sequence to k numbers, the decoder reconstructs twelve from those k numbers, and training adjusts the two together to make the reconstruction error smaller. There is no label; the target is the input itself.

# common.py — MODEL. The FICTIONAL sequence of `architectures/06` is built in its RAW form:
# subscriber level + periodic component + the residual of the earlier lessons.
import math

SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12


def generator(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_sequences(n, seed):
    D = []
    for i in range(n):
        r = generator(seed + i)
        r()
        residual = [0.0] + [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)]
        r()
        s = generator(seed + 6000 + i)
        level = 21.0 + (s() + s() + s() - 1.5) * 6.0
        a, b = (s() + s() - 1.0) * 2.6, (s() + s() - 1.0) * 2.6
        angle = [2 * math.pi * t / STEPS for t in range(STEPS)]
        D.append([round(level + a * math.sin(angle[t]) + b * math.cos(angle[t]) + residual[t], 3)
                  for t in range(STEPS)])
    return D


DATA = build_sequences(720, SEED)
TRAIN, TEST = DATA[:150], DATA[540:]
MEAN = [sum(x[t] for x in TRAIN) / len(TRAIN) for t in range(STEPS)]
SPREAD = [max(1e-9, (sum((x[t] - MEAN[t]) ** 2 for x in TRAIN) / len(TRAIN)) ** 0.5)
          for t in range(STEPS)]
ZTRAIN = [[(x[t] - MEAN[t]) / SPREAD[t] for t in range(STEPS)] for x in TRAIN]
ZTEST = [[(x[t] - MEAN[t]) / SPREAD[t] for t in range(STEPS)] for x in TEST]


def distance(u, v):
    return sum((u[t] - v[t]) ** 2 for t in range(STEPS)) ** 0.5


ZERO = [0.0] * STEPS
BASELINE = sum(distance(z, ZERO) for z in ZTEST) / len(ZTEST)
print(f"sequences {len(DATA)}, length {STEPS}, training {len(TRAIN)}, held-out {len(TEST)}")
print(f"example raw sequence {DATA[0]}")
print(f"baseline (reconstruction with the mean sequence) {BASELINE:.4f}")

sequences 720, length 12, training 150, held-out 180
example raw sequence [21.188, 21.247, 19.977, 20.223, 21.497, 22.188, 24.826, 24.397, 24.089, 25.549, 22.073, 22.529]
baseline (reconstruction with the mean sequence) 2.9007

Under the setup there are three components: the subscriber level and the two coefficients of the periodic component. Even if twelve numbers are stored, what the sequence carries adds only noise on top of these three. This is what the autoencoder is expected to find, and at which k it finds it can be measured.

def autoencoder(seed, k, epochs=300, lr=0.3):
    # the data is centered with the column means, so no bias term is written;
    # the k=0 case is exactly the baseline.
    r = generator(seed)
    g = lambda: (r() + r() + r() - 1.5) * 0.7
    W1 = [[g() for _ in range(STEPS)] for _ in range(k)]
    W2 = [[g() for _ in range(k)] for _ in range(STEPS)]
    n = len(ZTRAIN)
    for _ in range(epochs):
        gW1 = [[0.0] * STEPS for _ in range(k)]
        gW2 = [[0.0] * k for _ in range(STEPS)]
        for z in ZTRAIN:
            h = [sum(W1[i][t] * z[t] for t in range(STEPS)) for i in range(k)]
            y = [sum(W2[t][i] * h[i] for i in range(k)) for t in range(STEPS)]
            d = [2.0 * (y[t] - z[t]) / STEPS for t in range(STEPS)]
            for t in range(STEPS):
                for i in range(k):
                    gW2[t][i] += d[t] * h[i]
            for i in range(k):
                dh = sum(d[t] * W2[t][i] for t in range(STEPS))
                for t in range(STEPS):
                    gW1[i][t] += dh * z[t]
        for i in range(k):
            for t in range(STEPS):
                W1[i][t] -= lr * gW1[i][t] / n
        for t in range(STEPS):
            for i in range(k):
                W2[t][i] -= lr * gW2[t][i] / n
    return {"k": k, "W1": W1, "W2": W2}


def reconstruct(P, z):
    h = [sum(P["W1"][i][t] * z[t] for t in range(STEPS)) for i in range(P["k"])]
    return [sum(P["W2"][t][i] * h[i] for i in range(P["k"])) for t in range(STEPS)]


def error(P, Z):
    return sum(distance(z, reconstruct(P, z)) for z in Z) / len(Z)


def direction_difference(A, B):
    # distance between the normalized encoder directions of the two runs
    f = 0.0
    for i in range(A["k"]):
        u, v = A["W1"][i], B["W1"][i]
        nu = sum(q * q for q in u) ** 0.5
        nv = sum(q * q for q in v) ** 0.5
        f += min(sum(abs(u[t] / nu - v[t] / nv) for t in range(STEPS)),
                 sum(abs(u[t] / nu + v[t] / nv) for t in range(STEPS)))
    return f / A["k"]


print(f"baseline (k=0) {BASELINE:.4f}, epochs 300, training {len(ZTRAIN)}")
print(f"{'k':>3}{'parameters':>11}{'training 1':>12}{'training 2':>12}{'held-out 1':>12}"
      f"{'held-out 2':>12}{'run difference':>16}{'direction difference':>22}")
for k in (1, 2, 3, 4):
    A, B = autoencoder(SEED + 1, k), autoencoder(SEED + 2, k)
    a, b = error(A, ZTEST), error(B, ZTEST)
    print(f"{k:>3}{2 * STEPS * k:>11}{error(A, ZTRAIN):>12.4f}{error(B, ZTRAIN):>12.4f}"
          f"{a:>12.4f}{b:>12.4f}{abs(a - b):>16.4f}{direction_difference(A, B):>22.4f}")

baseline (k=0) 2.9007, epochs 300, training 150
  k parameters  training 1  training 2  held-out 1  held-out 2  run difference  direction difference
  1         24      1.3001      1.2999      1.2817      1.2815          0.0003                1.6330
  2         48      1.0322      1.0278      1.0597      1.0576          0.0021                2.8634
  3         72      0.7936      0.7502      0.7949      0.7638          0.0311                3.4005
  4         96      0.7219      0.7123      0.7346      0.7314          0.0031                3.3412

The baseline is 2.9007. A single latent number brings the error down to 1.2817 and three numbers to 0.7949; going from the third to the fourth the drop is about 0.06, because the structure the setup carries ends at three components and the fourth direction sits largely on noise. The Unsupervised Learning course had taken the same measure with computed directions; the autoencoder here does not compute the directions, it searches for them with gradient descent, and it goes below the baseline down to 0.7346.

The last two columns are the reading of this course. The error of the two runs is nearly the same in all four rows: the largest difference is 0.0311 at k=3. By contrast, the directions the same two runs found differ between 1.6330 and 3.4005 — for normalized directions of twelve components this is a large distance. The autoencoder finds a subspace, and that subspace is repeated from run to run; the axes inside the subspace are not repeated. Anyone who wants to name a latent dimension must know that what they name changes with the seed.

The Adversarial Generative Network and Instability

In the adversarial generative network there is no input to reconstruct. The generator builds a sequence from a latent vector, the discriminator tries to tell the real from the produced, and the loss of the generator is the discriminator being wrong. The goals of the two networks are the reverse of each other; for this reason the loss of neither side improves on its own, and the sentence “the loss went down” is not a sign of progress here.

LATENT, GEN_HIDDEN, DISC_HIDDEN, STOPS = 3, 6, 6, (30, 60, 90, 120)


def adversarial(seed, epochs=120, lr=0.6):
    r = generator(seed)
    g = lambda: (r() + r() + r() - 1.5) * 0.8
    GEN = {"W1": [[g() for _ in range(LATENT)] for _ in range(GEN_HIDDEN)],
           "b1": [[0.0] for _ in range(GEN_HIDDEN)],
           "W2": [[g() for _ in range(GEN_HIDDEN)] for _ in range(STEPS)],
           "b2": [[0.0] for _ in range(STEPS)]}
    DISC = {"W1": [[g() for _ in range(STEPS)] for _ in range(DISC_HIDDEN)],
            "b1": [[0.0] for _ in range(DISC_HIDDEN)], "w": [[g()] for _ in range(DISC_HIDDEN)],
            "b": [[0.0]]}
    z, N, trace = generator(seed + 77), len(ZTRAIN), []

    def generate(v):                    # generator: from latent vector to sequence
        h = [math.tanh(sum(GEN["W1"][i][j] * v[j] for j in range(LATENT)) + GEN["b1"][i][0])
             for i in range(GEN_HIDDEN)]
        return h, [sum(GEN["W2"][t][i] * h[i] for i in range(GEN_HIDDEN)) + GEN["b2"][t][0]
                   for t in range(STEPS)]

    def discriminate(x):                # discriminator: from sequence to a single number
        u = [sum(DISC["W1"][i][t] * x[t] for t in range(STEPS)) + DISC["b1"][i][0]
             for i in range(DISC_HIDDEN)]
        return u, sum(DISC["w"][i][0] * u[i] for i in range(DISC_HIDDEN) if u[i] > 0) + DISC["b"][0][0]

    def descend(P, grad):
        for key in P:
            for i in range(len(P[key])):
                for j in range(len(P[key][i])):
                    P[key][i][j] -= lr * grad[key][i][j]

    def zeros(P):
        return {key: [[0.0] * len(u) for u in P[key]] for key in P}

    for t in range(1, epochs + 1):
        Z = [[(z() + z() + z() - 1.5) * 1.6 for _ in range(LATENT)] for _ in range(N)]
        FAKE = [generate(v)[1] for v in Z]
        gD = zeros(DISC)
        for x, y in [(u, 1) for u in ZTRAIN] + [(u, 0) for u in FAKE]:
            u, o = discriminate(x)
            d = (1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - y) / (2 * N)
            gD["b"][0][0] += d
            for i in range(DISC_HIDDEN):
                gD["w"][i][0] += d * max(0.0, u[i])
                du = d * DISC["w"][i][0] if u[i] > 0 else 0.0
                gD["b1"][i][0] += du
                for j in range(STEPS):
                    gD["W1"][i][j] += du * x[j]
        gG = zeros(GEN)
        for v in Z:
            h, x = generate(v)
            u, o = discriminate(x)
            d = (1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - 1) / N
            dx = [sum(d * DISC["w"][i][0] * DISC["W1"][i][j] for i in range(DISC_HIDDEN) if u[i] > 0)
                  for j in range(STEPS)]
            for j in range(STEPS):
                gG["b2"][j][0] += dx[j]
                for i in range(GEN_HIDDEN):
                    gG["W2"][j][i] += dx[j] * h[i]
            for i in range(GEN_HIDDEN):
                dh = sum(dx[j] * GEN["W2"][j][i] for j in range(STEPS)) * (1 - h[i] * h[i])
                gG["b1"][i][0] += dh
                for j in range(LATENT):
                    gG["W1"][i][j] += dh * v[j]
        descend(DISC, gD)
        descend(GEN, gG)
        if t in STOPS:
            trace.append((t, sum(discriminate(x)[1] > 0 for x in ZTRAIN) / N,
                          sum(discriminate(x)[1] <= 0 for x in FAKE) / N))
    mean = [sum(x[t] for x in FAKE) / N for t in range(STEPS)]
    return trace, mean, [(sum((x[t] - mean[t]) ** 2 for x in FAKE) / N) ** 0.5
                         for t in range(STEPS)]

REAL_MEAN = [sum(x[t] for x in ZTRAIN) / len(ZTRAIN) for t in range(STEPS)]
REAL_SPREAD = [(sum((x[t] - REAL_MEAN[t]) ** 2 for x in ZTRAIN) / len(ZTRAIN)) ** 0.5
               for t in range(STEPS)]
RUNS = [adversarial(SEED + j) for j in (1, 2)]
print(f"{'epoch':>5}" + "".join(f"{'r' + str(j + 1) + ' real':>12}{'r' + str(j + 1) + ' fake':>11}"
                                for j in (0, 1)))
for i, (t, _real, _fake) in enumerate(RUNS[0][0]):
    print(f"{t:>5}" + "".join(f"{RUNS[j][0][i][1]:>12.4f}{RUNS[j][0][i][2]:>11.4f}"
                              for j in (0, 1)))


def deviation(a, b):
    return sum(abs(a[t] - b[t]) for t in range(STEPS)) / STEPS


print(f"{'measure':<30}{'run 1':>10}{'run 2':>10}{'difference between runs':>25}")
print(f"{'deviation of mean from real':<30}{deviation(RUNS[0][1], REAL_MEAN):>10.4f}"
      f"{deviation(RUNS[1][1], REAL_MEAN):>10.4f}{deviation(RUNS[0][1], RUNS[1][1]):>25.4f}")
print(f"{'deviation of spread from real':<30}{deviation(RUNS[0][2], REAL_SPREAD):>10.4f}"
      f"{deviation(RUNS[1][2], REAL_SPREAD):>10.4f}{deviation(RUNS[0][2], RUNS[1][2]):>25.4f}")

epoch     r1 real    r1 fake     r2 real    r2 fake
   30      0.6133     0.8067      0.5200     1.0000
   60      0.3933     0.7800      0.6267     1.0000
   90      0.3467     0.6600      0.5800     1.0000
  120      0.4600     0.8533      0.6400     0.9333
measure                            run 1     run 2  difference between runs
deviation of mean from real       0.5321    0.8877                   1.2707
deviation of spread from real     0.3669    0.5434                   0.3418

The first table shows the instability of the arrangement directly. In the first run the accuracy of the discriminator on the real sequences falls from 0.6133 to 0.3467 and then returns to 0.4600; meanwhile the rate of recognizing the generated ones falls from 0.8067 to 0.6600 and rises to 0.8533. In the second run the discriminator catches all of the generated ones in three of the measurements — there the generator made no progress at all. At which epoch one stops changes the result, and there is no one-directional curve that would tell the stopping point.

The second table is the closing number of this lesson. The step means produced by the two runs deviate from the real set by 0.5321 and 0.8877, but they deviate from each other by 1.2707. The distance between the two runs is larger than the distance of each to the target; that is, the two runs have gone not toward the same place but to two separate places. In the autoencoder the same comparison did not exceed 0.0311. The difference in stability comes not from the capacity of the arrangement but from the form of its target: minimizing a fixed target and fooling a moving opponent are not the same problem.

Summary

• There is a measure even without labels: the baseline of the autoencoder is reconstruction with the mean sequence and is 2.9007; this baseline is the case where the latent dimension is zero.
• One latent dimension brings the error down to 1.2817 and three dimensions to 0.7949; the fourth dimension adds only about 0.06, because the structure of the setup ends at three components.
• While the reconstruction error of the two runs differs by at most 0.0311, the directions they find differ between 1.6330 and 3.4005: the subspace is repeated, the axes are not.
• In the adversarial arrangement the accuracy of the discriminator on the real sequences moves between 0.6133 and 0.3467 and gives no one-directional curve; in the second run the generator makes no progress at all.
• The step means produced by the two runs differ from each other by 1.2707 — more than the deviation of each from the real set, 0.5321 and 0.8877.

Next Step

In ten lessons the architecture was chosen. Convolution used the grid structure of the input, recurrence and gated units the sequence structure, attention and multi-head attention direct access to distant steps, embedding the discrete symbols, transfer a representation from another task, and the generative arrangements an unlabeled target. The difference each added was measured against its baseline and the second run was written next to every number. All ten of these lessons have one thing in common that was never asked: all of them ran in a small training loop written by hand. The gradient was derived on this page every time, the data fit in memory, the batch size was equal to the number of samples, the seed stood in a single place and no record of any run was kept. The next topic takes up not the architecture but the setup of training: deriving the gradient from a computation graph instead of by hand, producing the data in batches, the relation of memory to batch size, the reproducible record of runs, and how the search for settings goes wrong with run-to-run variability.