Unit 03 of 03
Evaluation and Tuning
Separating out which measure, which threshold and how many candidates a model's number came from: the different error distribution of two models giving the same accuracy and the different cost of the two error types, the ranking changing across accuracy, precision, recall and F1, the opposite movement of precision and recall as the threshold is swept and which curve is misleading on an imbalanced set, the difference between the stated probability and the realized rate and the existence of a model whose ranking is good but whose calibration is bad, mean absolute error, mean squared error and the explained variation ratio not giving the same ranking, the estimate shifting as the number of folds grows and the number of trainings rising, the best candidate three searches find with the same budget and the validation number that inflates as the budget grows, the slice where the error concentrates and the number of bad slices that grows by chance as the slices multiply.
Published Updated
Lessons
- 01 Confusion Matrix Separating the error distributions of two models with the same accuracy: among forty-nine candidates, fourteen accuracy values are shared by more than one candidate, two candidates giving 0.7817 catch 16 of the 61 suspicious subscribers in one and 32 in the other, and the number of flagged subscribers ranges from 26 to 58. When a field visit counts as 1 and a missed illicit case as 6, the cost is 296 and 232, the baseline costs 366, and calling everyone suspicious at 252 is cheaper than the baseline. The best candidate by accuracy gives 263 in cost, while the best candidate by cost is twentieth among forty-nine candidates in accuracy. 14 min read
- 02 Classification Metrics Ranking five models by four metrics derived from the confusion matrix: the model that is first in accuracy is second in precision and fourth in recall and the F1 score, three different models take the lead on the four metrics, and seven of the ten model pairs swap places between accuracy and F1. Because the baseline flags no subscriber, its precision is undefined and its recall and F1 score are zero. For a procedure with precision 1.0000 and recall 0.0164, the arithmetic mean is 0.5082 while the F1 score is 0.0323, and the difference rises to 0.4759. 13 min read
- 03 Threshold Selection and Curves Separating the two numbers the threshold moves from the area under the curve: as the threshold goes from 0.10 to 0.90, precision rises from 0.3608 to 1.0000, recall falls from 0.9344 to 0.0328 and the cost rises from 182 to 356. The 0.16 chosen with the validation set among ninety-nine threshold candidates gives a cost of 181 on the test set, is 86 units cheaper than the default threshold of 0.50, and would have looked 7 units better if it had been read from the test set. When the suspicious share is lowered from 0.2421 to 0.0354, the area under the ROC curve rises from 0.8058 to 0.8624 while that of the precision–recall curve falls from 0.6212 to 0.4124. 11 min read
- 04 Probability Calibration Measuring the difference between the stated probability and the realized share: the calibration error of the linear model is 0.0242, that of the sharpened score, whose ranking is exactly equal to it, is 0.1512, and the area under the curve of the two models is also 0.8058. The calibration error of the depth 8 tree is 0.0000 on the training set by definition and 0.1435 on the test set; the calibration mapping learned from training converges to the identity mapping and corrects nothing. The single-variable mapping learned from the validation set lowers the error of the sharpened score from 0.1512 to 0.0463 and leaves the area under the curve at 0.8058. 5 min read
- 05 Regression Metrics The shift of the measure from counting to distance when the target is a number, and two members of the same family trading places on three measures: the linear model fitted to squared error leads on mean squared error with 22.3164, the one fitted to absolute error leads on mean absolute error with 3.6178. The baseline is the train mean (19.7665 m³) and the coefficient of determination is defined exactly relative to this baseline. 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; when those 13 rows are trimmed, the ranking reverses on both measures. The number of candidates tried is three and no hyperparameter was swept. 10 min read
- 06 Cross-Validation Measuring the number of folds itself: when k is swept from 2 to 10, the mean of the estimate stays between 0.8115 and 0.8204, the true number on the test set is 0.8056 and the closest estimate to the truth comes from k=4. Leave-one-out gives 0.8175, 0.0119 away from the truth, but for this it processes 1,008 trainings and 1,015,056 rows — 111.89 times k=10. Its per-fold spread is 0.3863 and this number is not uncertainty but the correct/incorrect scatter of a one-row fold; because none of the 1,008 folds contains both classes, precision and recall cannot be computed per fold. On the other hand, the dependence of leave-one-out on the shuffle is exactly zero. 13 min read
- 07 Hyperparameter Search The lesson where the rule of the course is paid: with the same budget of 50 candidates, the best candidate found by grid, random and model-based search and the number of that candidate on the test set are printed side by side. As the number of candidates rises from 1 to 200, the best number on the validation set climbs from 0.7937 to 0.8452, the number of the same candidate on the test set goes from 0.7778 to 0.7976, and the inflation between them opens from +0.0159 to +0.0476. The untuned default candidate already gives 0.8452 on the validation set, that is, 200 candidates found nothing. At budgets of 50 and 200 the single-split decision rule beats the linear model on the test set: the family difference is −0.0159 and −0.0040 while the inflation by the budget is +0.0516 and +0.0476. 14 min read
- 08 Error Analysis Breaking the 51 errors on the test set down by region, tariff step, household size and number of readings: in two of the fourteen measurable slices the model falls below the baseline, and the central region holds 23% of the rows while carrying 35.3% of the errors. As the number of slices goes from 14 to 41 and from 41 to 50, the number of bad slices found goes from 2 to 6 and from 6 to 9 — but when the same errors are distributed randomly over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, that is, the observed number is below chance in all three sets. The course wrap-up gathers the family, the number of candidates and the margin above the baseline of twenty lessons in a single table. 12 min read