Unit 02 of 03
Nonlinear Models
Counting the difference and the cost that the families which come into play when the decision boundary is curved add: the number that the neighbor count and the distance measure move and the nearest and farthest neighbors drawing closer as columns increase, the probability output running to the extremes when the conditional independence assumption breaks while the ranking survives, the margin width and the number of support vectors that carry the decision and the result the kernel transformation gives without being expanded, the split a splitting criterion chooses and how pruning differs from a depth limit, the number that settles and the variability that falls as the number of trees grows together with the out-of-bag estimate, the trade-off between learning rate and number of rounds in sequential boosting, the difference three combinations add and the number of examples on which the ensembles err together.
Published Updated
Lessons
- 01 Nearest Neighbor Measuring a family that computes nothing and stores the training set as it is: twenty-four candidates are tried over the neighbor count and the distance metric, the candidate chosen on the validation set gives 0.7937 on the test set and beats the baseline by 0.0357, yet the same candidate scores 0.8294 on the set it was chosen on. A procedure that looks at one neighbor falls below the baseline on all three metrics. As the column count grows, the ratio of the farthest neighbor to the nearest drops from 26.91 to 1.49 and the model settles exactly on the baseline. 14 min read
- 02 Probabilistic Classifier with an Independence Assumption Measuring a family that counts columns instead of summing them: instead of the 201,600 combinations of eight columns, a count table of 80 cells is kept, twenty candidates are tried and the candidate chosen on the validation set gives 0.7817 on the test set. A zero count eliminates a class entirely; with 60 training rows, 137 test rows are vetoed, and smoothing removes the veto but does not guarantee accuracy. When the column carrying the same information is added eight times, subscribers with extreme probabilities rise from 2 to 169 and the mean confidence from 0.842 to 0.970; the hits in the top 61 of the ranking go from 29 to 33. 14 min read
- 03 Margin-Based Classifier Measuring a family that bases its decision only on the examples near the boundary: as the penalty coefficient is swept, the margin widens from 0.047 to 93.488, the candidate chosen on the validation set gives 0.7857 on the test set and the rule that resolves the tie alone is worth four subscribers. The decision is carried by the 324 rows inside the margin; dropping 300 rows from outside the margin changes 4 predictions, dropping 300 rows from inside changes 21. Nine candidates are tried with three kernels and the curved boundary cannot beat the straight one; the polynomial kernel gives the same prediction on 252 of 252 test rows as 90 explicitly computed columns, without opening those 90 columns at all. 14 min read
- 04 Decision Trees Measuring the split chosen by the split criterion and how pruning differs from a depth limit: impurity and entropy choose the same root split but agree on only eighteen of forty-seven splits, while the misclassification rate chooses an entirely different 695/61 split. The unpruned depth-12 tree with 127 leaves gives 0.7341 on the test set, below the baseline; pruning brings the same tree down to 20 leaves and up to 0.7857. The depth limit gives 0.7897 with 48 leaves while pruning gives 0.7857 with 11 leaves, and four of those are deeper than the sixth level. The trees of eight 90% subsamples share not a single split. 14 min read
- 05 Bagged Tree Ensembles Counting the fragility of a single tree and measuring what bagging adds: across 41 bag samples the same procedure chooses nine different column-threshold pairs for the root split, two trees classify on average 46.2 of the 252 subscribers in the validation set differently, and the single tree's test number wanders between 0.6905 and 0.7897. As the number of trees goes from 1 to 41, the spread on the validation set falls from 0.0193 to 0.0075. When feature randomness is added, the ensemble chosen on the validation set among 31 candidates gives 0.7897 on the test set; the out-of-bag estimate, 0.7884, stands 0.0013 from this number, while the validation set's own number stands 0.0516 away. 9 min read
- 06 Sequential Boosting Measuring what happens when trees are built on the remaining error instead of independently of one another: over two hundred rounds the training number climbs from 0.7778 to 0.8638, while the validation number peaks at 0.8373 in round ten and drops to 0.8056. The learning rate and the number of rounds substitute for each other; the same target is reached in 79 rounds with a rate of 0.02, in 30 with 0.05, in 10 with 0.15 and in 4 with 0.45, and the product of rate and rounds stays between 1.50 and 1.80. On the seventy-two-cell grid, the candidate chosen from validation gives 0.7738 on the test set and the one chosen from the test set gives 0.8056: a difference of 0.0318. In total 1,279 candidates were tried, and boosting with 30 splits fell 0.0159 behind bagging with 568 splits. 8 min read
- 07 Ensemble Methods Counting side by side what bagging, boosting and stacking add and what they cost: the three base models give 0.7778, 0.7897 and 0.7738 on the test set at costs of 43, 568 and 30 splits. On 45 of the 252 subscribers all three are wrong, bagging and boosting classify only 6 subscribers differently, and no combination that sees nothing but the three labels can pass 0.7897. When the second layer of stacking is learned from the validation set it reports 0.8254 and gives 0.7778; when it is learned from the training set it reports 0.8532 and again gives 0.7778: the inflation goes from 0.0476 to 0.0754 and the weight shifts to the model that memorizes most. The topic's total tuning budget is 1,313 candidates. 12 min read