Skip to content
academia.sh
tr
Log in

Course Advanced

Deep Learning

Published Updated

By the end of this course

Start course

01

Neural Network Fundamentals

Measuring what each part of a fully connected network contributes and how much a second run of the same network moves: the single-layer network being the same thing as the linear model, what adding a layer changes, saturation shrinking the gradient and the dead unit count, the growth of parameter and multiplication counts as layers widen, the choice of loss giving two different results in the same network and the loss not being the same thing as the metric, backpropagation checked against the numerical derivative, the loss and stability reached by three optimizers within the same epoch budget, the vanishing and exploding gradient as depth grows, the contribution of dropout and early stopping and which set early stopping is read from, the learning rate that batch normalization allows.

  1. 01 Perceptron and Multilayer Network First measurement of whether a network's number comes from its capacity or from its training: initialized from zero, the single-layer network produces all three numbers of the Supervised Learning course's linear model at its three settings, down to the last digit. On the reduced 300-row training set, the two-layer network with 113 parameters reaches 0.7857 across four candidates, with a baseline of 0.7579; but a second run of the same network, with only the initial values changed, gives 0.7698, and the band of five runs is 0.7659-0.7857. Run-to-run variability is five subscribers, the difference capacity adds is one subscriber; the two-layer network whose activation is left linear cannot get out of the single-layer band despite its 113 parameters.
  2. 02 Activation Functions Measuring the choice of activation through saturation, dead units and the gradient that reaches the first layer: the derivative of the sigmoid falls from 0.1966 at z=1 to 0.0025 at z=6, a ratio of 79.7; for the hyperbolic tangent the ratio is 17,088. In the same network the mean gradient reaching the first layer is 1.0456 with the sigmoid and 4.4498 with the rectified linear unit — 4.26 times. Three activations with two runs each: sigmoid 0.7937 and 0.7976, rectified 0.7857 and 0.7698, hyperbolic tangent 0.7659 and 0.7698; baseline 0.7579. When the learning rate is raised to 20, four of the rectified units die, and at 30 all eight do, and the mean derivative becomes exactly 0.0000; with the sigmoid and the hyperbolic tangent no unit dies, but the mean derivative falls from 0.1971 to 0.0295.
  3. 03 Forward Propagation Measuring the width of the hidden layer with two cost numbers and one return number: in a network with 12 inputs, when the hidden layer grows from 2 to 32 units, the parameter count grows from 29 to 449 (15.5 times), the multiplications per example from 26 to 416 (16.0 times), and the total multiplications of a hundred-epoch training from 780,000 to 12,480,000. The return stays small beside this: the test band of ten runs is 0.7540-0.7857, that is, eight subscribers; the difference between the good runs of each width is five subscribers and the largest difference between two runs at a single width is also five subscribers. The width chosen from validation is 4 units, that is, 57 parameters, and the test number of that candidate is 0.7540, below the baseline; its second run is 0.7738.
  4. 04 Loss Functions Measuring how the choice of loss moves the gradient the network sees and the number it reaches: when the network says 0.99 and the true label is 0, the gradient that cross-entropy gives to the output layer is 0.9900 and that of squared error 0.0196 — a ratio of 50.5. When the same network is trained with two losses on four candidates, validation picks squared error and 25 epochs; the test number of that candidate is 0.7619, its second run 0.7897, a difference of seven subscribers. What the choice of loss adds is three subscribers. While the training loss always falls from 0.4861 to 0.3279 and the training accuracy rises from 0.78 to 0.86, the test accuracy peaks at 0.7937 at epoch 50 and falls to 0.7817; in the same run the number of suspicious subscribers caught rises from 1 to 25.
  5. 05 Backpropagation Applying the chain rule by hand over the network and verifying the resulting gradient with the numerical derivative: across all 113 parameters the largest absolute difference is 2.193e-10 at initialization and 6.441e-11 after a hundred epochs of training. When the step size is chosen as 0.001, the difference in the trained network jumps to 2.518e-01; this is not a calculation error, it is the kink of the rectified unit. The cost: backpropagation produces all 113 gradients with one forward and one backward pass per example, the numerical derivative asks for 4,520 passes for the same job, and the ratio grows with the parameter count: 58, 226, 898. The triple does not change — baseline 0.7579, network number 0.7857, second run 0.7698.
  6. 06 Optimization Algorithms Using the same gradient with three different step rules and measuring that a falling loss does not mean a falling error: in the shared 40-epoch budget momentum descent brings the training loss down to 0.2777 and the adaptive step to 0.2923, plain descent stays at 0.3296; the test accuracies, on the other hand, are 0.7500, 0.7222 and 0.7698. In the three settings read from validation the network number is 0.7857, the second run of the same network is 0.7698 and the baseline is 0.7579. When four runs are made changing only the initialization, the test spread of a single optimizer rises to 0.0317, that is, eight subscribers; the difference between optimizers, on the other hand, moves between 0.0516 and 0.0079 depending on the run, and none of the twelve runs passes the 0.8016 of the linear model.
  7. 07 Training Stability Measuring the vanishing and exploding of the gradient as it passes through the layers: with saturating activation the gradient magnitude of the first layer goes from 1.274e-02 with one hidden layer down to 5.539e-05 with six layers, that is, 230 times lower, and the ratio to the last layer rises from 18.8 to 3,977.2; with the rectified linear unit the ratio at the same depth stays at 0.7022, while when the initialization gain is raised to 16 it grows 134.99 times. With a learning rate of 10 the gradient magnitude of the network with three hidden layers shoots up to 2.800e+07, 15 of the 24 units die and both runs give exactly the baseline, 0.7579. Gradient clipping clips 400 of the 450 updates and saves the training, but the test falls to 0.6984, below the baseline; clipping and the schedule together bring the loss down to 0.2722 and raise the run difference to 0.0516, that is, thirteen subscribers.
  8. 08 Regularization Measuring the contribution of two network-specific regularization methods and the set from which early stopping is read: the gap between training and test of the bare network is 0.0887, with dropout at rate 0.4 it falls to 0.0330 and the test goes from 0.7579 to 0.7937; the second run of the same setting gives 0.7738 and there the bare network beats the dropout network. Weight decay had been swept in Introduction to Machine Learning and Supervised Learning, here it is applied with a single coefficient and brings the gap down to 0.0702. In early stopping the validation loss sees its bottom at the tenth and the fourteenth epoch; when the stopping epoch is read from training the test comes out as 0.7579 and 0.7817, from validation 0.7698 and 0.7976, from test 0.7738 and 0.8016 — the inflation from reading off the test is 0.0040 in both runs, that is, one subscriber.
  9. 09 Batch Normalization Measuring the learning rate that scaling the layer inputs within the mini-batch allows and the source of leakage it opens: the network without normalization carries the loss to 8.6577 at learning rate 20 and at 60 kills 14 of 16 units and gives exactly the baseline; the network with normalization holds the loss at 0.7206 and 2.3263 at the same rates and not a single unit dies at any rate — the largest rate that keeps the loss under 1.0 goes from 5 to 20, a fourfold increase. When inference is made with running averages the test is 0.7698, when it is made with the statistics of the test batch itself it is 0.7937; the inflation is 0.0238, that is, six subscribers. When only the order of the test rows is changed the same model gives 0.7817, seven subscribers change class and the probability of a single subscriber moves by 0.2273.

02

Image and Sequence Architectures

Counting what architectures that use the structure of the input add and what they still miss: how many weights convolution needs to do the same job as a fully connected layer, the information pooling throws away and its resistance to shifts, the gradient reaching the first layer as depth grows and the skip connection correcting it, how much information the hidden state can carry as the sequence lengthens, how many steps back the gated units carry a dependency, how many of the attention weights go to the correct step and why a high weight is not an explanation, the difference the number of heads adds and the order information that disappears when positional encoding is removed, the difference between a learned embedding and a projection that is not learned, how many examples transfer adds with and the distinction between freezing and fine-tuning, the reconstruction error and the instability of the adversarial arrangement between two runs.

  1. 01 Convolution Operation Counting how many weights the first layer that uses the structure of the input needs: a fully connected layer producing the same 144 outputs holds 20,880 numbers while a 3x3 convolution kernel holds 10, a ratio of 2,088 times. Two hand-written kernels give 0.8900 agreement without padding and 0.9380 with padding 1, and the difference is entirely the edge cells. On the test set the most frequent class gives 0.5850, while a fully connected network with 1,169 parameters memorizes the training set completely and reads 0.5100 and 0.5350, and a convolutional network with 23 parameters reads 0.8000 and 0.8150: the difference between the two runs is 0.0250 for the fully connected network and 0.0150 for the convolutional network.
  2. 02 Pooling Counting what pooling discards and what it gives in return: 2x2 max pooling reduces 144 cells to 36, 96.9 cells can be recovered on average and an uncertainty of 46.0 bits remains. When the raw grid is pooled the line information is lost entirely and agreement falls from 0.9380 to 0.4220; when the same pooling is applied to the feature map, shift invariance rises from 0.0% to 38.1%, and to 88.7% with global pooling. With a baseline of 0.5850, the network without pooling gives 0.5150 and 0.5350 with 309 parameters, the 2x2 pooled network gives 0.6100 and 0.6150 with 93 parameters, and the globally pooled network gives 0.8050 and 0.8050 with 23 parameters; the number of decisions that change on the shifted test set is 84, 49 and 12 respectively.
  3. 03 Convolutional Network Architectures Counting what depth adds and whether the gradient reaches the first layer: in the plain stack the magnitude of the gradient reaching the first layer falls from 6.405e-01 at depth 0 to 9.269e-07 at depth 8, the skip connection holds 6.779e-01 at the same depth and the ratio is 731,349 times. With a baseline of 0.5850, the shallow network with 23 parameters gives 0.7850 and 0.8050, and the plain depth 2 network with 63 parameters gives 0.5850 in both runs, that is, the deeper network cannot beat the shallower one. The skip connection gives 0.7950 and 0.8150 at the same depth and 0.8150 and 0.8900 at depth 5; the mean difference of 0.0575 that depth adds is smaller than the difference of 0.0750 between two runs of the same network.
  4. 04 Recurrent Networks Counting how much information the hidden state carries as the sequence grows longer: when only the first step of the same sequence is changed, the difference in the last hidden state is 0.92844 at length 2 and falls to 0.00913 at length 12, a factor of 0.62 per step. The accuracy of reading the marker at the first step back from the last state falls from 1.0000 to 0.5300, that is, to chance level. The same network with 61 parameters beats the baseline by 0.2300 and 0.2800 at length 2, by 0.2100 and 0.1700 at length 8 and by 0.1850 and 0.1850 at length 10, while at length 12 it beats it by only 0.0250 and 0.0050: the dependency is carried over ten steps and is not carried over twelve steps.
  5. 05 Gated Units Counting how many steps back a long dependency can be carried: in the untrained network the share of the marker at the first step that remains in the last hidden state is 0.98% and 0.05% in the plain network with 6 units, 0.71% and 0.05% in the plain network with 9 units, and 9.34% and 5.63% in the network with gated units. With the same epoch budget all three structures beat the baseline by 0.1700–0.2800 up to length 8; at length 12 the plain network with 61 parameters drops to 0.0250 and 0.0050, while the gated network with 115 parameters gives 0.2650 and 0.2050 and the wide plain network with 118 parameters gives 0.2700 and 0.1700. The advantage of the gated structure over the equal-budget plain network cannot be measured at twelve steps, because the difference of 0.1000 between the two runs of the wide plain network is larger than the difference between the two structures.
  6. 06 Attention Mechanism Measuring what looking directly at a distant step of the sequence gains and how much of the weights that are read amounts to an explanation: while the most-frequent-class baseline is 0.6333, two runs of the same attention network give 0.8889 and 0.9944 on the held-out set, and the 0.1056 between them comes from the initialization seed alone; the first run sends only 0.2721 of the weight to the marker step and puts the largest weight on the correct step in 0.3444 of the examples while producing 0.8889 accuracy, whereas the second run gives 0.9691 and 1.0000. In the step-removal test the agreement between weight and effect stays at 0.5889 in the first run; in the second run, in 43 of the 180 examples whose weight exceeds 0.5, removing that step moves the prediction by less than 0.05.
  7. 07 Multi-Head Attention and Transformers Measuring what multiplying the query and the positional encoding add under the same parameter budget: with 33 parameters held fixed, the held-out mean is 0.8722, 0.8806 and 0.8972 for 1, 2 and 4 heads, so the number of heads adds 0.0250 while the difference between the two runs of a single setup rises to 0.0500. The second run of the two-head network separates the heads to the two ends of the sequence (weight centers 3.30 and 8.55, separation 0.7264) but the first run does not (5.57 and 6.30) and its accuracy does not drop. When the positional encoding is removed, both runs fall exactly to the baseline, 0.6611, and when the readings are shuffled the output moves by 0.0000.
  8. 08 Embedding Layers Measuring what mapping a discrete event code to a continuous vector adds: while the most-frequent-class baseline is 0.6444, the learned embedding with 103 parameters gives 0.9722 and 0.9833 at 150 epochs, the one-hot encoding with 157 parameters stays at 0.7167 and 0.7000 in the same epochs and reaches the same figure only at 600 epochs (0.9611 and 0.9556). The frozen table of the same size stays at 0.6944 and 0.7278, and the ordinal numeric code stands exactly at the baseline in all four epoch budgets. In the learned table the distance between the suspect codes and the other codes is 3.9153 times the within-group distance; in the frozen table it is 1.0728 times.
  9. 09 Transfer Learning Measuring what carrying a pretrained model, produced by our own hands from a fictional source task, over to a target task adds: while the most-frequent-class baseline is 0.6444, training from scratch with twenty labels gives 0.7194, transfer with freezing 0.8417 and transfer with fine-tuning 0.8083. Freezing stays flat at around 0.83 in all four measurements and training from scratch passes it between forty and eighty examples; fine-tuning stays above training from scratch at three of the four points, but at one hundred sixty examples the difference falls to 0.0028. The table from the source task carries a maintenance separation of 3.6373 and a suspect separation of 2.4767; after fine-tuning these two numbers become 2.4685 and 3.7991.
  10. 10 Generative Models Measuring two generative arrangements that work without labels on the same fictional sequence set: while the baseline of reconstruction with the mean sequence is 2.9007, the linear autoencoder gives 1.2817 and 1.2815 with one latent dimension and 0.7346 and 0.7314 with four; the error of the two runs differs by at most 0.0311, while the directions the same runs found differ between 1.6330 and 3.4005. In the adversarial arrangement the accuracy of the discriminator on the real sequences falls from 0.6133 to 0.3467 and returns to 0.4600, and the step means produced by the two runs move 1.2707 away from each other — more than the deviation of each from the real set, 0.5321 and 0.8877.

03

Training Practice

Measuring how much the training setup moves the number: the gradient difference between the computation graph and automatic differentiation and the hand-written backpropagation and the code each one takes, the difference data augmentation adds and the case where augmentation produces leakage, the batch generation order moving the result, the relation of batch size to the number of stored values for parameters, activations and gradients and the need for it to change together with the learning rate, the items to pin that are specific to training and how much the result moves when one of them is released, what the search selects when run-to-run variability is larger than the difference between candidates.

  1. 01 Computation Graph and Automatic Differentiation Writing, within the lesson, the abstraction that separates gradient computation from the architecture and comparing it with a hand-written backpropagation: a thirty-two-line machinery with four operations gives the gradient of eighty-one parameters with a difference of 2.78e-17 from the eighteen-line hand-written backpropagation and builds 6,622 nodes for a batch of thirty examples. The course's three numbers are set up here: the baseline 0.7579, the network's number 0.8095 and the second run, which differs only in the initial values, 0.7937. The range of ten initializations is between 0.7857 and 0.8095, and the numbers of the Supervised Learning course's decision tree (0.7857) and linear model (0.8016) fall within this range.
  2. 02 Data Loading Pipeline Measuring the batch generation order, data augmentation and the leakage augmentation produces separately: with the same network and the same 400 updates, four batch orders give four numbers between 0.8016 and 0.8095 on the test set, while the difference rises to 0.0159 when only the initial values change under the same order — the order decision moves half as much as run-to-run variability. The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the run range of a single order. When augmentation is done before the split, the twins of 138 rows of the test set enter the training set and the twinned-clean subset gap shifts from −0.0211 to −0.0008: a shift of +0.0203.
  3. 03 Hardware Usage Computing the relation between the batch size and the number of stored values in memory, and measuring that the batch size must change together with the learning rate: a one-layer eight-unit network holds 1,182 values at batch size 30 and 0.8629 of them are activations; in a four-layer sixty-four-unit network the share drops to 0.0198 at batch size 1 and rises to 0.8583 of 185,242 values at batch size 300. In a budget of one hundred thousand values the largest batch size is 2,936, 676 and 139 respectively. If the learning rate is left at 0.30 while the number of epochs is constant, the spread of four batch sizes becomes 0.0476 (twelve subscribers) and the lowest, 0.7619, stays only 0.0040 above the baseline; when the step is scaled with the batch size, the spread falls to 0.0159.
  4. 04 Experiment Tracking Counting the items that must be pinned for the same training to give the same number a second time: on top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course and the 11 of the Visualization, Reporting and Analysis at Scale course, 8 training-specific items are added, making 67. Six of the eight items move the result when released alone, and the largest shift is 0.0159, four subscribers — that item is not a setting of the model but its starting values. The summation order moves the parameters by 6.66e-16 and changes no decision, while the number of epochs moves the parameters by 0.2098 and again changes no decision; the number of epochs, which does not move the result alone, shifts it by two subscribers when released together with the batch size.
  5. 05 Hyperparameter Search Measuring how run-to-run variability misleads a search: when ten candidates are run twice with the same budget, the spread between candidates on the validation set is 0.0317 while the difference between two runs of a single candidate reaches 0.0238, and twenty-three of the forty-five candidate pairs cannot be told apart, in seven of them the order flips. The search looking from the first run chooses the sixteen-unit candidate and gets 0.8135 on the test set, while the search looking from the second run chooses the four-unit candidate and gets 0.8016; the test mean of the ten candidates is 0.8036 and the baseline is 0.7579. The course wrap-up gathers the baseline, the network's number and the second run of twenty-four lessons in a single table.

Final exam

Deep Learning — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close