Unit 02 of 03
Image and Sequence Architectures
Counting what architectures that use the structure of the input add and what they still miss: how many weights convolution needs to do the same job as a fully connected layer, the information pooling throws away and its resistance to shifts, the gradient reaching the first layer as depth grows and the skip connection correcting it, how much information the hidden state can carry as the sequence lengthens, how many steps back the gated units carry a dependency, how many of the attention weights go to the correct step and why a high weight is not an explanation, the difference the number of heads adds and the order information that disappears when positional encoding is removed, the difference between a learned embedding and a projection that is not learned, how many examples transfer adds with and the distinction between freezing and fine-tuning, the reconstruction error and the instability of the adversarial arrangement between two runs.
Published Updated
Lessons
- 01 Convolution Operation Counting how many weights the first layer that uses the structure of the input needs: a fully connected layer producing the same 144 outputs holds 20,880 numbers while a 3x3 convolution kernel holds 10, a ratio of 2,088 times. Two hand-written kernels give 0.8900 agreement without padding and 0.9380 with padding 1, and the difference is entirely the edge cells. On the test set the most frequent class gives 0.5850, while a fully connected network with 1,169 parameters memorizes the training set completely and reads 0.5100 and 0.5350, and a convolutional network with 23 parameters reads 0.8000 and 0.8150: the difference between the two runs is 0.0250 for the fully connected network and 0.0150 for the convolutional network. 10 min read
- 02 Pooling Counting what pooling discards and what it gives in return: 2x2 max pooling reduces 144 cells to 36, 96.9 cells can be recovered on average and an uncertainty of 46.0 bits remains. When the raw grid is pooled the line information is lost entirely and agreement falls from 0.9380 to 0.4220; when the same pooling is applied to the feature map, shift invariance rises from 0.0% to 38.1%, and to 88.7% with global pooling. With a baseline of 0.5850, the network without pooling gives 0.5150 and 0.5350 with 309 parameters, the 2x2 pooled network gives 0.6100 and 0.6150 with 93 parameters, and the globally pooled network gives 0.8050 and 0.8050 with 23 parameters; the number of decisions that change on the shifted test set is 84, 49 and 12 respectively. 13 min read
- 03 Convolutional Network Architectures Counting what depth adds and whether the gradient reaches the first layer: in the plain stack the magnitude of the gradient reaching the first layer falls from 6.405e-01 at depth 0 to 9.269e-07 at depth 8, the skip connection holds 6.779e-01 at the same depth and the ratio is 731,349 times. With a baseline of 0.5850, the shallow network with 23 parameters gives 0.7850 and 0.8050, and the plain depth 2 network with 63 parameters gives 0.5850 in both runs, that is, the deeper network cannot beat the shallower one. The skip connection gives 0.7950 and 0.8150 at the same depth and 0.8150 and 0.8900 at depth 5; the mean difference of 0.0575 that depth adds is smaller than the difference of 0.0750 between two runs of the same network. 8 min read
- 04 Recurrent Networks Counting how much information the hidden state carries as the sequence grows longer: when only the first step of the same sequence is changed, the difference in the last hidden state is 0.92844 at length 2 and falls to 0.00913 at length 12, a factor of 0.62 per step. The accuracy of reading the marker at the first step back from the last state falls from 1.0000 to 0.5300, that is, to chance level. The same network with 61 parameters beats the baseline by 0.2300 and 0.2800 at length 2, by 0.2100 and 0.1700 at length 8 and by 0.1850 and 0.1850 at length 10, while at length 12 it beats it by only 0.0250 and 0.0050: the dependency is carried over ten steps and is not carried over twelve steps. 13 min read
- 05 Gated Units Counting how many steps back a long dependency can be carried: in the untrained network the share of the marker at the first step that remains in the last hidden state is 0.98% and 0.05% in the plain network with 6 units, 0.71% and 0.05% in the plain network with 9 units, and 9.34% and 5.63% in the network with gated units. With the same epoch budget all three structures beat the baseline by 0.1700–0.2800 up to length 8; at length 12 the plain network with 61 parameters drops to 0.0250 and 0.0050, while the gated network with 115 parameters gives 0.2650 and 0.2050 and the wide plain network with 118 parameters gives 0.2700 and 0.1700. The advantage of the gated structure over the equal-budget plain network cannot be measured at twelve steps, because the difference of 0.1000 between the two runs of the wide plain network is larger than the difference between the two structures. 14 min read
- 06 Attention Mechanism Measuring what looking directly at a distant step of the sequence gains and how much of the weights that are read amounts to an explanation: while the most-frequent-class baseline is 0.6333, two runs of the same attention network give 0.8889 and 0.9944 on the held-out set, and the 0.1056 between them comes from the initialization seed alone; the first run sends only 0.2721 of the weight to the marker step and puts the largest weight on the correct step in 0.3444 of the examples while producing 0.8889 accuracy, whereas the second run gives 0.9691 and 1.0000. In the step-removal test the agreement between weight and effect stays at 0.5889 in the first run; in the second run, in 43 of the 180 examples whose weight exceeds 0.5, removing that step moves the prediction by less than 0.05. 12 min read
- 07 Multi-Head Attention and Transformers Measuring what multiplying the query and the positional encoding add under the same parameter budget: with 33 parameters held fixed, the held-out mean is 0.8722, 0.8806 and 0.8972 for 1, 2 and 4 heads, so the number of heads adds 0.0250 while the difference between the two runs of a single setup rises to 0.0500. The second run of the two-head network separates the heads to the two ends of the sequence (weight centers 3.30 and 8.55, separation 0.7264) but the first run does not (5.57 and 6.30) and its accuracy does not drop. When the positional encoding is removed, both runs fall exactly to the baseline, 0.6611, and when the readings are shuffled the output moves by 0.0000. 15 min read
- 08 Embedding Layers Measuring what mapping a discrete event code to a continuous vector adds: while the most-frequent-class baseline is 0.6444, the learned embedding with 103 parameters gives 0.9722 and 0.9833 at 150 epochs, the one-hot encoding with 157 parameters stays at 0.7167 and 0.7000 in the same epochs and reaches the same figure only at 600 epochs (0.9611 and 0.9556). The frozen table of the same size stays at 0.6944 and 0.7278, and the ordinal numeric code stands exactly at the baseline in all four epoch budgets. In the learned table the distance between the suspect codes and the other codes is 3.9153 times the within-group distance; in the frozen table it is 1.0728 times. 14 min read
- 09 Transfer Learning Measuring what carrying a pretrained model, produced by our own hands from a fictional source task, over to a target task adds: while the most-frequent-class baseline is 0.6444, training from scratch with twenty labels gives 0.7194, transfer with freezing 0.8417 and transfer with fine-tuning 0.8083. Freezing stays flat at around 0.83 in all four measurements and training from scratch passes it between forty and eighty examples; fine-tuning stays above training from scratch at three of the four points, but at one hundred sixty examples the difference falls to 0.0028. The table from the source task carries a maintenance separation of 3.6373 and a suspect separation of 2.4767; after fine-tuning these two numbers become 2.4685 and 3.7991. 13 min read
- 10 Generative Models Measuring two generative arrangements that work without labels on the same fictional sequence set: while the baseline of reconstruction with the mean sequence is 2.9007, the linear autoencoder gives 1.2817 and 1.2815 with one latent dimension and 0.7346 and 0.7314 with four; the error of the two runs differs by at most 0.0311, while the directions the same runs found differ between 1.6330 and 3.4005. In the adversarial arrangement the accuracy of the discriminator on the real sequences falls from 0.6133 to 0.3467 and returns to 0.4600, and the step means produced by the two runs move 1.2707 away from each other — more than the deviation of each from the real set, 0.5321 and 0.8877. 11 min read