Unit 03 of 03
Training Practice
Measuring how much the training setup moves the number: the gradient difference between the computation graph and automatic differentiation and the hand-written backpropagation and the code each one takes, the difference data augmentation adds and the case where augmentation produces leakage, the batch generation order moving the result, the relation of batch size to the number of stored values for parameters, activations and gradients and the need for it to change together with the learning rate, the items to pin that are specific to training and how much the result moves when one of them is released, what the search selects when run-to-run variability is larger than the difference between candidates.
Published Updated
Lessons
- 01 Computation Graph and Automatic Differentiation Writing, within the lesson, the abstraction that separates gradient computation from the architecture and comparing it with a hand-written backpropagation: a thirty-two-line machinery with four operations gives the gradient of eighty-one parameters with a difference of 2.78e-17 from the eighteen-line hand-written backpropagation and builds 6,622 nodes for a batch of thirty examples. The course's three numbers are set up here: the baseline 0.7579, the network's number 0.8095 and the second run, which differs only in the initial values, 0.7937. The range of ten initializations is between 0.7857 and 0.8095, and the numbers of the Supervised Learning course's decision tree (0.7857) and linear model (0.8016) fall within this range. 12 min read
- 02 Data Loading Pipeline Measuring the batch generation order, data augmentation and the leakage augmentation produces separately: with the same network and the same 400 updates, four batch orders give four numbers between 0.8016 and 0.8095 on the test set, while the difference rises to 0.0159 when only the initial values change under the same order — the order decision moves half as much as run-to-run variability. The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the run range of a single order. When augmentation is done before the split, the twins of 138 rows of the test set enter the training set and the twinned-clean subset gap shifts from −0.0211 to −0.0008: a shift of +0.0203. 14 min read
- 03 Hardware Usage Computing the relation between the batch size and the number of stored values in memory, and measuring that the batch size must change together with the learning rate: a one-layer eight-unit network holds 1,182 values at batch size 30 and 0.8629 of them are activations; in a four-layer sixty-four-unit network the share drops to 0.0198 at batch size 1 and rises to 0.8583 of 185,242 values at batch size 300. In a budget of one hundred thousand values the largest batch size is 2,936, 676 and 139 respectively. If the learning rate is left at 0.30 while the number of epochs is constant, the spread of four batch sizes becomes 0.0476 (twelve subscribers) and the lowest, 0.7619, stays only 0.0040 above the baseline; when the step is scaled with the batch size, the spread falls to 0.0159. 12 min read
- 04 Experiment Tracking Counting the items that must be pinned for the same training to give the same number a second time: on top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course and the 11 of the Visualization, Reporting and Analysis at Scale course, 8 training-specific items are added, making 67. Six of the eight items move the result when released alone, and the largest shift is 0.0159, four subscribers — that item is not a setting of the model but its starting values. The summation order moves the parameters by 6.66e-16 and changes no decision, while the number of epochs moves the parameters by 0.2098 and again changes no decision; the number of epochs, which does not move the result alone, shifts it by two subscribers when released together with the batch size. 9 min read
- 05 Hyperparameter Search Measuring how run-to-run variability misleads a search: when ten candidates are run twice with the same budget, the spread between candidates on the validation set is 0.0317 while the difference between two runs of a single candidate reaches 0.0238, and twenty-three of the forty-five candidate pairs cannot be told apart, in seven of them the order flips. The search looking from the first run chooses the sixteen-unit candidate and gets 0.8135 on the test set, while the search looking from the second run chooses the four-unit candidate and gets 0.8016; the test mean of the ten candidates is 0.8036 and the baseline is 0.7579. The course wrap-up gathers the baseline, the network's number and the second run of twenty-four lessons in a single table. 7 min read