Perceptron and Multilayer Network

Across twenty-one lessons, eleven of them unlabeled, every method worked on the assumption that the features were given. Scaling, encoding, generation and reduction were always decided by hand, and the cost of each decision was measured separately. What a procedure that learns these decisions from the data itself would gain, and what it would make invisible, was never asked.

The neural network is a claimed answer to that question: instead of designing the intermediate representation by hand, determine the weights of the hidden layers by training as well. This lesson builds the smallest form of that claim and sets up the three numbers of the course: the baseline, the network’s number, and the second run of the same network, with only the initial values changed. The third is the signature of the course, because a network’s number can come from its training rather than its capacity. One more thing is announced up front: on this fictional tabular data the network is not expected to beat the decision tree or the linear model, and this will not be hidden.

• NN1. The dataset, the split and the seed carry over unchanged from the Introduction to Machine Learning, Data Preparation and Feature Engineering, Supervised Learning and Unsupervised Learning courses: the fictional subscriber table, 756/252/252, seed 20260218. The target is the suspicious label and the baseline is saying the most frequent class.
• NN2. The feature set is the twelve columns left by the Data Preparation and Feature Engineering course and is fixed throughout the course. The network takes these columns as given; the scaling, encoding and derivation decisions are still made by hand.
• NN3. The columns are scaled with the mean and dispersion learned from the training set. The Data Preparation and Feature Engineering course had measured that this step added +0.0437 to a gradient-based model; here it is assumed.
• NN4. In this course the training set is reduced to 300 rows. The reason is run time: every lesson trains the same network at least twice. The cost of the reduction is measured in this lesson.
• NN5. The solution is gradient descent; its procedure was built in the Linear Models topic of the Supervised Learning course and is not repeated here. How the gradient is distributed across layers is the subject of the Backpropagation lesson.
• NN6. The weights are started with gain-scaled initialization: the starting range is scaled with the square root of the number of inputs. The biases start from zero.
• NN7. The metric is accuracy, the test set is 252 subscribers and one subscriber is worth 0.0040. The number of candidates tried is 4, and the choice is made on the validation set.

Weight, Bias and Unit

A unit is built from three parts: each input gets a weight, a constant is added to the weighted sum, and the result is passed through an activation function. In this course the added constant is called the bias. The distinction is necessary: the bias in the bias–variance trade-off of the Introduction to Machine Learning course is an error component, whereas the bias here is a learned parameter and shifts the threshold of the unit.

Units standing side by side form a layer. This lesson compares two architectures: the single-layer network that connects the input directly to the output, and the two-layer network that carries a layer of eight units in between.

# neural_net.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split;
# the target is the label of suspected illicit use or fault.
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):                                 # self-written generator, no library
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS}
STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS}
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)


def scaled(x):                              # FIT on train, APPLY to every set
    return [(x[a] - MEAN[a]) / STD[a] for a in FIELDS]


def make_set(data):
    return [scaled(x) for x in data], [float(x["suspicious"]) for x in data]


X_FULL, Y_FULL = make_set(TRAIN)            # full training set, 756 rows
X_TR, Y_TR = make_set(TRAIN[:300])          # REDUCED training set, 300 rows
X_VA, Y_VA = make_set(VAL)
X_TE, Y_TE = make_set(TEST)

print(f"train {len(X_FULL)}, reduced train {len(X_TR)}, validation {len(X_VA)}, "
      f"test {len(X_TE)}, columns {len(FIELDS)}")
print(f"suspicious rate in full train {sum(Y_FULL) / len(Y_FULL):.4f}, "
      f"in reduced {sum(Y_TR) / len(Y_TR):.4f}, in test {sum(Y_TE) / len(Y_TE):.4f}")
print(f"baseline (most frequent class) on test {BASELINE:.4f}")

train 756, reduced train 300, validation 252, test 252, columns 12
suspicious rate in full train 0.2222, in reduced 0.2200, in test 0.2421
baseline (most frequent class) on test 0.7579

The comparison line comes from the Supervised Learning course and was measured with the full 756 rows: the gradient-based linear model and the decision tree tuned on validation give 0.8016 on the test set, and the untuned decision tree 0.7857. What the network has to beat is not the baseline but this number.

A Single-Layer Network Is Not a New Model

A network with no hidden layer carries twelve weights and one bias, feeds the weighted sum to an S-shaped function and produces a probability. This description is word for word the same as that of the logistic regression of the Supervised Learning course, and the sameness can be tested: if the weights are started from zero and trained with the same learning rate and number of epochs, the network must reproduce the table of that lesson.

def squash(z):                              # S-shaped function, overflow-protected
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def init_weights(seed, sizes):              # gain-scaled initialization, bias zero
    r, net = rng(seed), []
    for a, b in zip(sizes, sizes[1:]):
        scale = math.sqrt(2.0 / a)
        net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b))
    return net


def forward(net, x, linear=False):          # FORWARD PASS: layer by layer
    a, acts = x, [x]
    for k, (W, bias) in enumerate(net):
        z = [sum(w * v for w, v in zip(s, a)) + b for s, b in zip(W, bias)]
        a = [squash(z[0])] if k == len(net) - 1 else (
            z[:] if linear else [t if t > 0 else 0.0 for t in z])
        acts.append(a)
    return acts


def train_model(net, X, Y, rate, epochs, linear=False):   # MODEL: full-batch gradient descent
    n = len(X)
    for _ in range(epochs):
        gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net]
        gb = [[0.0] * len(bias) for _, bias in net]
        for i in range(n):
            acts = forward(net, X[i], linear)
            d = [acts[-1][0] - Y[i]]
            for k in range(len(net) - 1, -1, -1):
                W, a = net[k][0], acts[k]
                for j, dj in enumerate(d):
                    if dj:
                        s = gW[k][j]
                        for p in range(len(a)):
                            s[p] += dj * a[p]
                        gb[k][j] += dj
                if k:
                    prev = [0.0] * len(a)
                    for j, dj in enumerate(d):
                        if dj:
                            s = W[j]
                            for p in range(len(a)):
                                prev[p] += dj * s[p]
                    d = prev if linear else [prev[p] if a[p] > 0 else 0.0
                                             for p in range(len(a))]
        for k, (W, bias) in enumerate(net):
            for j in range(len(W)):
                s, g = W[j], gW[k][j]
                for p in range(len(s)):
                    s[p] -= rate * g[p] / n
                bias[j] -= rate * gb[k][j] / n
    return net


def accuracy(net, X, Y, linear=False):
    return sum((forward(net, x, linear)[-1][0] > 0.5) == (y > 0.5)
               for x, y in zip(X, Y)) / len(X)


def param_count(sizes):
    return sum(a * b + b for a, b in zip(sizes, sizes[1:]))


print(f"single-layer network parameter count {param_count([12, 1])}, "
      f"two-layer (8 units) {param_count([12, 8, 1])}")
print(f"{'learning rate':>14} {'epoch':>5} {'train':>9} {'validation':>10} {'test':>8}")
for rate, epochs in ((0.1, 100), (0.1, 300), (0.1, 600)):
    net = train_model([([[0.0] * 12], [0.0])], X_FULL, Y_FULL, rate, epochs)   # from zero
    print(f"{rate:>14} {epochs:>5} {accuracy(net, X_FULL, Y_FULL):>9.4f} "
          f"{accuracy(net, X_VA, Y_VA):>10.4f} {accuracy(net, X_TE, Y_TE):>8.4f}")

single-layer network parameter count 13, two-layer (8 units) 113
 learning rate epoch     train validation     test
           0.1   100    0.8175     0.8056   0.8016
           0.1   300    0.8201     0.8373   0.7937
           0.1   600    0.8188     0.8373   0.8016

All nine numbers of the three rows are the same, down to the last digit, as the first three rows of the logistic regression table of the Supervised Learning course. The single-layer network is not a new model family but a known model written in another vocabulary. Nothing the name “neural network” adds appears in these rows.

What Makes a Layer Nonlinear

The hidden layer brings 113 parameters, against 13 for the single-layer network. But capacity is not only the number of parameters. If the activation of the hidden layer is left as the identity function, that is, if the weighted sum is passed through as it is, the composition of two matrix multiplications is again a single matrix multiplication. The set of functions the network can represent does not change; the same function is merely written with more parameters.

The measurement shows this: three architectures, the same setting, two runs. The activation of the hidden layer is once the identity function and once the rectified linear unit, the function that cuts everything below zero to zero.

ARCHITECTURES = (("single-layer", [12, 1], False),
                 ("two-layer linear", [12, 8, 1], True),
                 ("two-layer rectified", [12, 8, 1], False))
print(f"{'architecture':<24} {'params':>9} {'run':>6} {'train':>9} "
      f"{'validation':>10} {'test':>8}")
BAND = {}
for name, sizes, lin in ARCHITECTURES:
    BAND[name] = []
    for run in (1, 2):
        net = train_model(init_weights(SEED + run, sizes), X_TR, Y_TR, 0.3, 100, lin)
        s = accuracy(net, X_TE, Y_TE, lin)
        BAND[name].append(s)
        print(f"{name:<24} {param_count(sizes):>9} {run:>6} "
              f"{accuracy(net, X_TR, Y_TR, lin):>9.4f} {accuracy(net, X_VA, Y_VA, lin):>10.4f} {s:>8.4f}")
print()
for name, v in BAND.items():
    print(f"{name:<24} test band {min(v):.4f}-{max(v):.4f}")

architecture                params    run     train validation     test
single-layer                    13      1    0.8433     0.7976   0.7817
single-layer                    13      2    0.8300     0.8016   0.7698
two-layer linear               113      1    0.8233     0.8056   0.7698
two-layer linear               113      2    0.8267     0.8016   0.7738
two-layer rectified            113      1    0.8400     0.8214   0.7857
two-layer rectified            113      2    0.8367     0.8056   0.7698

single-layer             test band 0.7698-0.7817
two-layer linear         test band 0.7698-0.7738
two-layer rectified      test band 0.7698-0.7857

The network with linear activation stays inside the band of the single-layer network despite its 113 parameters: 0.7698–0.7738 against 0.7698–0.7817. Eight times the parameters is not enough to produce a separate number. Nonlinear separation comes not from the number of layers but from the activation function; 0.7857 is the only number that rises above the others, and it belongs to the network with rectified units.

The Signature of the Course: The Second Run

The upper end, 0.7857, is a number that can be reported on its own, and when it is reported it misleads. The same network, the same data, the same setting, with only the initial values changed, is trained five times.

CANDIDATES = []
print(f"{'rate':>5} {'epoch':>5} {'validation':>10} {'test':>8}")
for rate in (0.3, 0.5):
    for epochs in (100, 200):
        net = train_model(init_weights(SEED + 1, [12, 8, 1]), X_TR, Y_TR, rate, epochs)
        CANDIDATES.append((rate, epochs, accuracy(net, X_VA, Y_VA), accuracy(net, X_TE, Y_TE)))
        print(f"{rate:>5} {epochs:>5} {CANDIDATES[-1][2]:>10.4f} {CANDIDATES[-1][3]:>8.4f}")
CHOSEN = max(CANDIDATES, key=lambda t: t[2])
print(f"candidates tried {len(CANDIDATES)}, chosen (rate {CHOSEN[0]}, epochs {CHOSEN[1]})\n")

RUNS = {}
print(f"{'run':>6} {'single-layer':>14} {'two-layer':>14}")
for run in range(1, 6):
    row = []
    for sizes in ([12, 1], [12, 8, 1]):
        net = train_model(init_weights(SEED + run, sizes), X_TR, Y_TR, CHOSEN[0], CHOSEN[1])
        row.append(accuracy(net, X_TE, Y_TE))
    RUNS[run] = row
    print(f"{run:>6} {row[0]:>14.4f} {row[1]:>14.4f}")

single, two = [RUNS[k][0] for k in RUNS], [RUNS[k][1] for k in RUNS]
print(f"\nbaseline {BASELINE:.4f}, network {two[0]:.4f}, "
      f"second run {two[1]:.4f}, difference {round(abs(two[0] - two[1]) * len(X_TE))} subscribers")
for name, v in (("single-layer", single), ("two-layer", two)):
    print(f"{name:<13} band {min(v):.4f}-{max(v):.4f}, run-to-run variability "
          f"{max(v) - min(v):.4f} = {round((max(v) - min(v)) * len(X_TE))} subscribers")
print(f"capacity adds {max(two) - max(single):.4f} "
      f"(subscribers: {round((max(two) - max(single)) * len(X_TE))}); cost of the reduction "
      f"(0.8016 at 756 rows) {0.8016 - max(single):.4f} "
      f"(subscribers: {round((0.8016 - max(single)) * len(X_TE))})")

 rate epoch validation     test
  0.3   100     0.8214   0.7857
  0.3   200     0.8135   0.7817
  0.5   100     0.8175   0.7857
  0.5   200     0.7937   0.7738
candidates tried 4, chosen (rate 0.3, epochs 100)

   run   single-layer      two-layer
     1         0.7817         0.7857
     2         0.7698         0.7698
     3         0.7738         0.7738
     4         0.7698         0.7817
     5         0.7698         0.7659

baseline 0.7579, network 0.7857, second run 0.7698, difference 4 subscribers
single-layer  band 0.7698-0.7817, run-to-run variability 0.0119 = 3 subscribers
two-layer     band 0.7659-0.7857, run-to-run variability 0.0198 = 5 subscribers
capacity adds 0.0040 (subscribers: 1); cost of the reduction (0.8016 at 756 rows) 0.0199 (subscribers: 5)

The last two lines set the rule of the course. The difference that capacity adds is one subscriber, while the movement of the same capacity from run to run is five subscribers. The network’s 0.7857 comes not from capacity but from the initial values of that run; the fifth run gives 0.7659, and that number is only two subscribers above the baseline. The band of the single-layer network is narrower, three subscribers, because the problem it solves has a single best point; when a hidden layer is added, that guarantee disappears.

The second claim can also be read from the table. The best run of the network, 0.7857, equals the untuned decision tree and is four subscribers below the 0.8016 of the two families of the Supervised Learning course. Part of the gap comes from the reduction of the training set: the same single-layer network gives 0.8016 with 756 rows and 0.7817 with 300. The rest is about the data itself: in a subscriber table with twelve columns, there is little intermediate representation to learn.

Summary

• A unit is built from a weight, a bias and an activation function; the bias is a layer parameter and is separate from bias, the error component of the Introduction to Machine Learning course.
• The network with no hidden layer, initialized from zero, produces all nine numbers of the linear model of the Supervised Learning course down to the last digit: 0.8016, 0.7937 and 0.8016.
• If the activation of the hidden layer is left as the identity function, the network with 113 parameters cannot get out of the band of the one with 13; nonlinear separation comes from the activation, not from the number of layers.
• The triple: baseline 0.7579, network’s number 0.7857, second run 0.7698. The band of the five runs is 0.7659–0.7857; run-to-run variability is five subscribers, the difference capacity adds is one subscriber.
• The network stays at 0.7857: equal to the untuned decision tree, four subscribers below the 0.8016 of the two families of the Supervised Learning course; five subscribers of the gap come from reducing the training set to 300 rows.

Next Step

It was measured that nonlinear separation comes from the activation, but the two functions used, the rectified linear unit that cuts everything below zero and the S-shaped squashing at the output, were not chosen with any justification. The next lesson treats that choice not as a preference but as a number: a squashing function flattens at the ends and its gradient falls to nearly zero, while a cutting function silences completely a unit whose input stays below zero. How many of the eight units are never activated on any example and by how many times saturation shrinks the gradient: three activations side by side, each with two runs.