Unit 01 of 02
Preparation Steps
Counting how much each decision taken on the existing columns moves the number on the held-out set while the model stays fixed: the contribution of the cleaning decision and its opposite signs in two model families, in which model family standardization and normalization produce a contribution and why the scale is fitted only on the training set, the column count and leakage of one-hot, ordinal and target encoding, the three separate numbers of imputation, the missingness indicator and dropping the row, the result that oversampling, undersampling and class weight give on an imbalanced set, the opposite effect of the same outlier on a gradient-based and a tree-based method.
Published Updated
Lessons
- 01 Data Cleaning Measuring on the held-out set what a cleaning decision adds to the model: the three defects in the raw record are fixed one at a time, the model is held fixed, and two numbers on the test set are printed side by side. Fixing the unit mix-up gains the distance-based method twelve subscribers and costs the decision tree two; when all three defects are fixed together, the tree's test accuracy rises from 0.7659 to 0.7778 and the distance-based method's from 0.7540 to 0.7857, and the baseline is 0.7579 for both. 14 min read
- 02 Feature Scaling Measuring the standardization and normalization decisions separately in three model families: standardization adds +0.0437 to the gradient-based linear model, gives −0.0159 on the distance-based method, and its contribution on the decision tree is exactly zero. The scale is an object fitted on the training set; applied to the test set, two values fall outside the [0, 1] range and standard scores rise as high as 5.11. When all eleven columns are scaled, the distance-based method loses one more subscriber. 10 min read
- 03 Categorical Encoding Measuring three encodings as column count and contribution when a categorical field is turned into numbers: one-hot encoding the region raises test accuracy from 0.7302 to 0.7778, and the 240-code reading route field gives +0.0119 for 231 extra columns when one-hot encoded and the same +0.0119 with a single column when ordinal encoded. Target encoding inflates training accuracy to 0.9114 while dropping the test to 0.7421; computed in-fold, 0.0595 of the inflation on the training set recedes and the test returns to 0.7540. 14 min read
- 04 Missing Value Strategies Measuring three strategies for a column with missing values separately on the held-out set: median imputation gives 0.7619, adding a missingness indicator next to the imputation gives 0.8333, and dropping the missing rows from the training set gives 0.7659; the contribution of the indicator is +0.0714, that is, eighteen subscribers. The indicator also exceeds by 0.0555 the upper bound of 0.7778 given by the column with nothing missing, because the missingness itself carries information that the imputed value does not: the suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did. 13 min read
- 05 Imbalanced Classes Reading three answers to class imbalance with two measures at once: the baseline gives 0.7579 in accuracy while the number of suspicious subscribers it catches is zero; the model on the raw training set gives 0.7778 accuracy and catches 21 of 61 suspicious subscribers. Oversampling the minority twice lowers accuracy to 0.7619 and raises the caught count to 33, undersampling the majority gives 0.7143 and 36, and class weight 3.5 gives 0.6349 and 39. Oversampling the minority twice and class weight 2.0 produce the same prediction for all 252 subscribers in the test set. 12 min read
- 06 Outlier Management Measuring that a single clipping decision gives opposite signs in two model families: clipping with a fence learned from the training set adds +0.0040 to the gradient-based linear model on real extremes only and +0.0317 when the stuck meter records are also present; the same decision gives −0.0278 and −0.0357 on the decision tree. The suspicious rate of the 34 subscribers outside the fence is 0.6765, so clipping also erases real information, and 41 values are clipped without deleting a single row. 13 min read