Unit 01 of 03
Evaluation Design
Designing the measurement itself and writing down which subgroup every number holds for: the inflated number produced when the same subscriber lands in two sets and the group-based split that undoes it, the same model making two different operating points and two different decisions under two cost matrices, how many baselines a problem has and how the choice of baseline makes the same model read two different ways, how much narrower a margin the paired comparison gives for the difference between two models than the independent comparison, the difference between the training distribution and the production distribution reduced to a single number, and covariate shift and label shift calling for different fixes.
Published Updated
Lessons
- 01 Validation Strategies Measuring the same number separately on the whole and on thirteen subgroups, and testing whether the held-out set was truly held out: a depth-6 decision tree beats the baseline on the test set by 0.0198 yet falls below the baseline in five of the subgroups, and the gap among the twelve subgroups with an n above 25 comes out at 0.3576. The same subscriber landing in both sets in the 3,199-row reading table splits 416 subscribers across them and raises the difference above the baseline at depth 12 from 0.0108 to 0.1000; the group-based split undoes this 0.0892 inflation. The inflation corrupts not only the overall number but the gap as well: the subgroup gap that reads 0.1718 in the row split is 0.3846 in the group-based split. 7 min read
- 02 Metric Selection The same model taking two separate operating points and making two separate decisions under two cost matrices: in the structure where inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged, in the structure where illicit use is costly the threshold is 0.25 and 77 are flagged; the decision of 60 subscribers flips between the two structures. The baseline moves with the cost as well — flagging nobody is 0.2421 in one, flagging everybody is 0.7579 in the other. A single threshold spreads the gain across subgroups from +0.4138 to -0.4468, the gap is 0.8606 and in four of twelve subgroups the model stays below its own baseline; choosing a subgroup-specific threshold lowers the test cost in only four of them. 6 min read
- 03 Baseline Model Measuring that the baseline is not a tool but a design object: the same depth-6 decision tree reads between -0.0119 and +0.5357 against eight baseline candidates, and the choice alone moves the number by 0.5476. A lookup-table baseline has a capacity of its own — with five cells it ties the model at 0.7579, with fifteen cells it beats the model at 0.7897, with thirty-nine cells it drops to 0.7500. The choice of baseline changes direction in the subgroups too: the model that falls below the most-frequent-class baseline in four of twelve subgroups falls below the region-and-tier lookup table in eight, and the difference gap shrinks from 0.2173 to 0.1351. The baselines chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table. 6 min read
- 04 Statistical Significance Measuring the margin of the difference between two models and counting how much narrower a margin the paired comparison gives than the independent comparison: the margins of all the differences between the depth-6 decision tree and its four rivals contain zero, and only the 0.5357 difference stays outside. On the same 252 subscribers the paired margin runs from 0.0476 to 0.0794 and the independent margin from 0.1190 to 0.1310; the more the two models agree, the narrower the margin. As the set grows from 63 to 504, the paired margin falls from 0.1111 to 0.0556. Among the subgroups, the margin excludes zero for only four of thirteen differences, and +0.1389, one of the largest subgroup contributions of the course, is not one of them. 10 min read
- 05 Distribution Shift Reducing the difference between training and production data to a single number and placing that number next to the drop in performance: a discriminator model reads a shift score of 0.1172 under covariate shift and the model's difference above the baseline falls from 0.0198 to 0.0066, but under label shift the same indicator stays at 0.0063 while the suspicious rate climbs from 0.2421 to 0.4865. The two shifts call for two separate fixes: prior correction raises accuracy from 0.6306 to 0.6622 under label shift while changing no decision under covariate shift, and retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and drops label shift to 0.6261. On the time-ordered production set the subgroup gap falls from 0.3576 to 0.2573, and in none of the twelve subgroups does the model stay below the baseline. 6 min read