Unit 02 of 02
Generalization
Measuring what a model will do on data it has not seen and how this measure gets corrupted: the reasoning behind separating the training, validation and test sets and how many times the test set may be looked at, the point where the training and held-out numbers diverge as capacity rises, how much the same procedure swings across different training samples, the two numbers moving in opposite directions as the penalty coefficient grows and reading the best point from the validation set, and how much four kinds of leakage inflate the held-out number.
Published Updated
Lessons
- 01 Training, Validation and Test Sets Measuring that the same model gives two different numbers on data it has seen and data it has not seen: 1,260 subscribers are split 756/252/252, a depth 6 decision tree gives 0.8466 accuracy on training, 0.7897 on validation and 0.7778 on test, and the baseline is also three different numbers on the three sets: 0.7778, 0.8135, 0.7579. As selection is made among 48 candidates produced by the same procedure, the gap between the number on the selection set and the number on the unseen set grows from 0.0211 to 0.0411. On a time-ordered split the test accuracy drops to 0.6865, but the difference above the baseline grows from 0.0198 to 0.0397. 14 min read
- 02 Overfitting and Underfitting Measuring how the training and held-out numbers diverge as capacity is raised step by step: as the depth of the decision tree goes from 1 to 14, the training accuracy rises from 0.7778 to 0.9352, the held-out accuracy peaks at 0.8214 and falls to 0.7302, and the difference between the two numbers widens from -0.0357 to +0.2050. Only four of the eleven capacities can match the baseline of 0.8135 on the held-out set. When the training set is raised from 100 to 756 rows, the difference falls from 0.2219 to 0.1680; when the minimum leaf size is raised from 2 to 80, it falls from 0.1680 to -0.0198 and the model turns into exactly the baseline. 14 min read
- 03 Bias–Variance Tradeoff Splitting the error on the held-out set into two components: on training sets built by sampling 30 times with replacement, the total squared error comes to 41.242 for the baseline, 29.768 for the depth 3 tree and 33.810 for the depth 12 tree, in m³ squared; the variance component rises from 0.040 to 4.409 while the bias and noise component falls from 41.202 to 28.448 and stops there. The bias that capacity can remove is 12.754 m³ squared, and the remaining 28.4 m³ squared cannot be removed by any capacity. For the same subscriber, the mean range of the predictions given by 30 models is 3.587 m³ at depth 1 and 8.172 m³ at depth 12. 5 min read
- 04 Regularization Intuition Measuring how penalizing complexity moves two numbers in opposite directions: in a linear model with 29 columns on 120 rows, when the penalty coefficient is raised from 0.001 to 1000 the weight magnitude falls from 10.040 to 0.601 and the training error worsens from 4.603 to 5.453 m³, while the validation error falls from 5.924 to 5.553 and returns to 5.932. Decomposed by sampling with replacement, the variance falls from 12.988 to 0.181, while the bias hits a floor at 30.976 and jumps to 35.295. The best penalty read from validation is 100 and gives 5.559 m³ on the test set; the baseline is 6.556 and the unpenalized model 5.841 m³. 14 min read
- 05 Data Leakage Measuring how sources other than learning inflate the held-out number: against the honest line, which gives an accuracy of 0.7778 on the test set with a baseline of 0.7579, scaling before the split inflates this number by 0.0119, a feature that sees the future by 0.0754, a duplicated record by 0.0542 and a feature derived from the target by 0.1429; all four have the same sign. A model whose features have been replaced with six random columns reads an accuracy of 0.8757 on the held-out set while the duplicates are present and beats the baseline of 0.7857 by 9 points; on the rows whose twin it has not seen, it stays 9.2 points below the baseline. At the course wrap-up, the baseline, training and held-out numbers of eleven lessons are gathered in a single table. 14 min read