Unit 02 of 02
Feature Generation and Selection
Counting what generating new columns, dropping columns and binding the preparation to an object add on the held-out set: the individual contribution of ratio, difference, interaction and threshold indicators and those whose contribution comes out zero, the number of numbers that come out of a date and a short text, the number of features chosen by filter, wrapper and embedded methods and the inflation produced when the selection is made from the whole set, the relation between the number of components and the held-out number and what is lost from interpretability, the number of steps learned from the whole set without a pipeline, the two cross-validation numbers given by preparing outside the folds and inside the folds.
Published Updated
Lessons
- 01 Feature Generation Counting one by one what new columns derived from existing columns add on the held-out set: the ratio feature carries the test number from 0.7778 to 0.7897 while the difference and the indicator learned from the training set add 0.0040, the hand-set 25 m³ threshold loses 0.0159 and the product of two columns loses 0.0238, and the 40 m³ threshold produces exactly zero contribution. When the six columns are added cumulatively the training number rises from 0.8466 to 0.8677 while the held-out number peaks at 0.7897 and falls to 0.7698; the three columns that each add a positive contribution on their own add a total contribution of 0.0000 when added together. 14 min read
- 02 Time and Text Features Extracting five numbers each from a date and from a short text and counting their total contributions on the held-out set: while the line before the decision is 0.7897, the five numbers from the date lose 0.0198, the five numbers from the note add 0.0317, and the ten numbers together add 0.0198. The whole contribution comes from a single indicator; the seal trace indicator alone adds 0.0357 while the text length loses 0.0357 and the word count loses 0.0516. How many numbers a field yields does not determine the contribution; the relation of the field to the label does. 9 min read
- 03 Feature Selection Counting how many features the filter, wrapper and embedded methods select from the same 25 candidate columns, and the inflation produced by the selection itself: while the 25 columns without selection give 0.8135 on the test set, the filter gives 0.8056 with 10 columns, the wrapper 0.7579 with 4 columns and the embedded method 0.8254 with 14 columns. The wrapper reports 0.8413 on the validation set it reads its score from and drops to 0.7579 on the test set; the inflation is 0.0834. When the selection from 1,200 noise columns is made from the whole set, the held-out number inflates by 0.0198, 0.0040 and 0.0714, with the same sign in all three measurements. 14 min read
- 04 Dimensionality Reduction as a Feature Tool Measuring what happens to the held-out number and to interpretability when a component-based representation replaces thirteen raw features: while the raw columns give 0.8254 on the test set, two components give 0.8175, three components 0.8056, five components 0.7778 and eight components 0.7460, and no number of components catches up with the raw representation. The component carrying the largest component share falls below the baseline on its own with 0.7381, while the fifth component with a share of 0.093 gives 0.7659. In the raw representation 0.35 of the splits are read from the mean consumption, while in the component representation the same share falls on a mixture with no name. 13 min read
- 05 Pipelines Binding the preparation chain to one object and counting how many steps leak without a pipeline: when the four-step chain is applied by hand, all four are learned from the whole set and the test number rises from 0.7817 to 0.8294, an inflation of 0.0476. When imputation and scaling leak, the number does not change at all; when target encoding leaks it inflates by 0.0397, and when selection leaks as well it inflates by 0.0476. The pipeline learns 131 numbers, and all of them come from the training set; the contribution of the scaling step in the tree model is exactly zero, while the contribution of the target encoding step is minus 0.0317. 15 min read
- 06 Fitting with Cross-Validation Comparing the two cross-validation numbers that come from learning the preparation chain outside the folds and inside the folds with the actual number on the held-out set: while the actual number on the test set is 0.7817, preparation inside the folds gives 0.7966 and preparation outside the folds gives 0.8105, and the deviation of preparation inside the folds is 0.0149 against 0.0288 for the one outside. The moment the second step of the chain, target encoding, moves outside the folds, the number jumps from 0.7966 to 0.8115. The 0.8254 given by the single split equals the highest of the four folds and is 0.0437 away from the actual number. In the course wrap-up, the before-decision and after-decision numbers of the twelve lessons are gathered in a single table. 8 min read