Hardware Usage

The previous lesson held the batch size at thirty. This number is tied to a resource: how many examples’ forward and backward passes can stay in memory at the same time in one update. An accelerator is a class of hardware, and the subject of this lesson is not the properties of any product but the computation that is the same on every accelerator: as the batch size grows, which quantity grows by how much.

The second question stands in the shadow of the first. The batch size looks like a memory decision, but it also changes the size of the step gradient descent takes: the gradient of a batch is the mean of that batch, and as the batch grows the mean calms down. The lesson’s measure is two numbers: how much the batch size grows the number of stored values, and how much the test number drops when the batch size is changed alone.

• TP12. The dataset, the network, the baseline and the backpropagation are inherited from Lesson 01; the number of epochs is held constant at 40 in this lesson and the number of updates changes accordingly as the batch size changes.
• TP13. The measure is the number of stored values, not bytes. How many bytes a value takes is a type decision and was measured in the memory lesson of the Data Analysis with Python course: there the same column took 173,837, 19,251 and 3,628 bytes by representation, and when the width was reduced from 8 bytes to 1 byte the memory shrank 31.4 times while the total shifted by 6.9 m³. That computation is not repeated; it is taken as input.
• TP14. The three stacks counted are: the parameters, the gradients of the parameters and the activations of every example in the batch together with their gradients. Activations are stored for the backward pass; if they were not stored, the forward pass would have to run again.
• TP15. The optimizer’s own state is counted separately: gradient descent with momentum asks for one more value per parameter. This lesson runs with plain descent and the state is zero.
• TP16. The memory budget is a fictional number (100,000 values) and is not the memory of any product; its purpose is to show the computation from the budget to the batch size.
• TP17. The scaled step rule is directly proportional to the batch size: the step of 0.30 for a batch of thirty becomes 3.00 for a batch of three hundred. This is not a rule but a candidate, and it is measured.

The Dataset, the Network and the Baseline

# COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through
# Unsupervised Learning courses is generated with the same seed (20260218);
# the training set is reduced from 756 to 300 rows. The network and the backpropagation
# are inherited from Lesson 01; the number of epochs is 40 and constant.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]
FIELDS = ["area_south", "area_east", "area_west", "area_central",
          "household", "satisfaction", "period", "mean"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def mix(items, seed):
    r, s = rng(seed), list(items)
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


DATA = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() < 0.046:
        continue
    subscriber_id = 10001 + i
    r, v = rng(SEED + 7000 + subscriber_id), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            b[2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id)
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3]
         + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": household, "satisfaction": rating, "period": len(v),
         "mean": round(mean, 2), "suspicious": int(z > 1.35)}
    for q in REGION[1:]:
        x["area_" + q[0]] = int(b[0] == q[0])
    DATA.append(x)

K = mix(DATA, SEED + 90000)
TRAIN, TEST = K[:300], K[1008:]


def table(rows, stats=None):
    X = [[float(s[a]) for a in FIELDS] for s in rows]
    if stats is None:                     # scaling is learned from the training set only
        stats = []
        for j in range(4, 8):
            m = sum(v[j] for v in X) / len(X)
            stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X))))
    for v in X:
        for j, m, sd in stats:
            v[j] = (v[j] - m) / sd
    return X, [s["suspicious"] for s in rows], stats


X_train, y_train, STATS = table(TRAIN)
X_test, y_test, _ = table(TEST, STATS)
BASELINE = sum(v == 0 for v in y_test) / len(y_test)

G, H = len(FIELDS), 8


def init_weights(seed, h=H):             # gain-scaled initialization
    r = rng(seed)
    return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(h * G)] + [0.0] * h \
        + [(r() * 2 - 1) * math.sqrt(2.0 / h) for _ in range(h)] + [0.0]


def gradient(p, X, y, idx, h=H):         # the hand-written backpropagation of Lesson 01
    g = [0.0] * len(p)
    for k in idx:
        x, hh = X[k], [0.0] * h
        for j in range(h):
            s = p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G))
            hh[j] = s if s > 0 else 0.0
        s = p[-1] + sum(p[h * G + h + j] * hh[j] for j in range(h))
        d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k]
        g[-1] += d
        for j in range(h):
            g[h * G + h + j] += d * hh[j]
            if hh[j] > 0:
                g[h * G + j] += d * p[h * G + h + j]
                for i in range(G):
                    g[j * G + i] += d * p[h * G + h + j] * x[i]
    return [v / len(idx) for v in g]


def train_model(X, y, seed, batch=30, step=0.30, epochs=40, h=H):
    p = init_weights(seed, h)
    for t in range(epochs):
        perm = mix(range(len(X)), SEED + 700 + t)
        for b in range(0, len(perm) - batch + 1, batch):
            p = [a - step * v for a, v in zip(p, gradient(p, X, y, perm[b:b + batch], h))]
    return p


def accuracy(p, X, y, h=H):
    n = 0
    for k in range(len(X)):
        x = X[k]
        o = p[-1] + sum(p[h * G + h + j] * max(0.0, p[h * G + j]
                        + sum(p[j * G + i] * x[i] for i in range(G)))
                        for j in range(h))
        n += (o >= 0.0) == (y[k] == 1)
    return n / len(X)


print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, "
      f"features {G}, hidden units {H}, parameters {len(init_weights(SEED))}")
print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}")

fictional: train 300 rows (reduced from 756), test 252, features 8, hidden units 8, parameters 81
baseline (predicting the most frequent class, on the test set) 0.7579

How Many Values Are Held in One Update

The number of parameters is independent of the batch size; the number of activations is directly proportional to the batch size. The ratio of the two quantities depends on the shape of the network and the table reads it on three networks.

# The NUMBER OF STORED VALUES. Three stacks are held for an update: the parameters, their
# gradients, and the activations of every example in the batch (the intermediate values of
# the forward pass are stored for the backward pass) together with the gradients of those activations.
def count_values(g, h, layers, batch, state=0):
    params = g * h + h + (layers - 1) * (h * h + h) + h + 1
    return params, params * (1 + state), 2 * batch * (g + layers * h + 1)


print(f"{'layers':>7} {'hidden':>6} {'batch':>6} {'parameters':>10} {'gradient':>8} "
      f"{'activations':>14} {'total':>9} {'activation share':>16}")
for layers, h in ((1, 8), (1, 64), (4, 64)):
    for batch in (1, 30, 300):
        p, e, a = count_values(G, h, layers, batch)
        print(f"{layers:>7} {h:>6} {batch:>6} {p:>10} {e:>8} {a:>14} {p + e + a:>9} "
              f"{a / (p + e + a):>16.4f}")

p, e, a = count_values(G, H, 1, 30)
print(f"\nthis lesson's network (1 layer, {H} units, batch size 30): {p + e + a} values; "
      f"at batch size 300 {sum(count_values(G, H, 1, 300)):,}")
print(f"descent with momentum asks for one more value per parameter for the optimizer "
      f"state: {sum(count_values(G, H, 1, 30, 1))} values")
BUDGET = 100000
print(f"\nif the budget were {BUDGET:,} values, the largest batch size:")
for layers, h in ((1, 8), (1, 64), (4, 64)):
    params = count_values(G, h, layers, 1)[0]
    print(f"  layers {layers}, units {h:>2}: {(BUDGET - 2 * params) // (2 * (G + layers * h + 1)):>6,}")

 layers hidden  batch parameters gradient    activations     total activation share
      1      8      1         81       81             34       196           0.1735
      1      8     30         81       81           1020      1182           0.8629
      1      8    300         81       81          10200     10362           0.9844
      1     64      1        641      641            146      1428           0.1022
      1     64     30        641      641           4380      5662           0.7736
      1     64    300        641      641          43800     45082           0.9716
      4     64      1      13121    13121            530     26772           0.0198
      4     64     30      13121    13121          15900     42142           0.3773
      4     64    300      13121    13121         159000    185242           0.8583

this lesson's network (1 layer, 8 units, batch size 30): 1182 values; at batch size 300 10,362
descent with momentum asks for one more value per parameter for the optimizer state: 1263 values

if the budget were 100,000 values, the largest batch size:
  layers 1, units  8:  2,936
  layers 1, units 64:    676
  layers 4, units 64:    139

Three groups of rows show three different arrangements. In the one-layer eight-unit network the activation share is 0.1735 at batch size 1; it rises to 0.8629 at batch size 30 and 0.9844 at batch size 300. This network is so small that data fills almost all of the memory. In the four-layer sixty-four-unit network the table turns around: at batch size 1 the share is 0.0198, that is, the memory belongs to the parameters and gradients; but when the batch size goes up to 300 the share becomes 0.8583 in the same network and the total grows from 26,772 to 185,242, more than sixfold. The number of parameters did not change at all between these two rows.

The useful side of the computation appears when it is read in reverse. When a memory budget is given, the largest batch size can be solved: in a fictional budget of a hundred thousand values the small network can take 2,936 examples, the wide one-layer network 676 and the four-layer network 139. The same budget, the same hardware, a twentyfold difference. The batch size must be chosen not as a habit but as the result of this split; and the chosen number does not come alone, as the next section shows.

Batch Size and Learning Rate Change Together

# The batch size and the learning rate change TOGETHER. The number of epochs is constant at 40;
# as the batch size grows, the number of updates falls. The scaled step is directly
# proportional to the batch size.
print(f"{'batch':>6} {'updates':>11} {'fixed step 0.30':>25} {'scaled step':>25}")
FIXED, SCALED = [], []
for batch in (10, 30, 100, 300):
    a = [accuracy(train_model(X_train, y_train, SEED + k, batch, 0.30), X_test, y_test) for k in (1, 2)]
    o = 0.30 * batch / 30
    b = [accuracy(train_model(X_train, y_train, SEED + k, batch, o), X_test, y_test) for k in (1, 2)]
    FIXED += a
    SCALED += b
    print(f"{batch:>6} {40 * (len(X_train) // batch):>11} {a[0]:>13.4f} {a[1]:>6.4f}"
          f"   {'step ' + format(o, '.2f'):>12} {b[0]:>6.4f} {b[1]:>6.4f}")
print(f"spread over four batch sizes, fixed step  {max(FIXED) - min(FIXED):.4f} "
      f"({round((max(FIXED) - min(FIXED)) * len(X_test))} subscribers)")
print(f"same spread, scaled step                  {max(SCALED) - min(SCALED):.4f} "
      f"({round((max(SCALED) - min(SCALED)) * len(X_test))} subscribers)")
print(f"baseline {BASELINE:.4f}; lowest with fixed step {min(FIXED):.4f} "
      f"({min(FIXED) - BASELINE:+.4f}), lowest with scaled step {min(SCALED):.4f} "
      f"({min(SCALED) - BASELINE:+.4f})")

 batch     updates           fixed step 0.30               scaled step
    10        1200        0.7619 0.7857      step 0.10 0.8056 0.7937
    30         400        0.8095 0.7937      step 0.30 0.8095 0.7937
   100         120        0.7897 0.8056      step 1.00 0.8016 0.8016
   300          40        0.7619 0.7738      step 3.00 0.8056 0.7937
spread over four batch sizes, fixed step  0.0476 (12 subscribers)
same spread, scaled step                  0.0159 (4 subscribers)
baseline 0.7579; lowest with fixed step 0.7619 (+0.0040), lowest with scaled step 0.7937 (+0.0357)

The left column shows the cost of changing the batch size alone. With the step left at 0.30, the spread of the four batch sizes is 0.0476, twelve subscribers: three times run-to-run variability (four subscribers). There are two separate reasons at the two ends. At batch size 300 only forty updates are made and the network stops before training is finished; the result is 0.7619, only 0.0040 above the baseline, that is, one subscriber. At batch size 10 the number of updates rises to 1,200 but since each gradient is the mean of only ten examples it is noisy; a step of 0.30 stays too large together with that noise and the result again falls to 0.7619.

The right column scales the step with the batch size: 0.10, 0.30, 1.00 and 3.00. The spread of the four batch sizes falls to 0.0159, four subscribers — that is, the same size as the difference between two runs of the same setup. The lowest number rises from 0.7619 to 0.7937 and the difference above the baseline from 0.0040 to 0.0357. The batch size alone is not a hyperparameter: when it is changed the learning rate must change too, otherwise what is measured is not the memory decision itself but the step forgotten beside that decision.

Summary

• The values held in one update are three stacks: the parameters, their gradients, and the activations of the examples in the batch with their gradients; the first two are independent of the batch size, the third is directly proportional.
• A one-layer eight-unit network holds 1,182 values at batch size 30 and 0.8629 of them are activations; in the four-layer sixty-four-unit network the share is 0.0198 at batch size 1 and 0.8583 at batch size 300.
• Going from batch size 1 to 300 in the same network carries the total from 26,772 to 185,242; the number of parameters does not change at all.
• In a fictional budget of a hundred thousand values the largest batch size is 2,936, 676 and 139 for the three networks; the same budget produces a twentyfold difference.
• If the step is left at 0.30 while the number of epochs is constant, the spread of the four batch sizes is 0.0476 (twelve subscribers) and the lowest, 0.7619, stays one subscriber above the baseline; when the step is scaled with the batch size the spread comes to 0.0159 and the lowest number to 0.7937.

Next Step

The number of items pinned over three lessons grew quickly: seed, example order, batch size, learning rate, number of epochs, augmentation scale, from which set the scaling statistics are read. None of them is the model itself and all of them move the result; in this lesson, leaving a single item — the step — in place lowered the number by twelve subscribers. How many items need to be pinned for a training to give the same number a second time was never counted. The next lesson addresses experiment tracking: the items to pin are counted on top of the lists of the earlier courses, and how much the result moves when one of them is released is printed.