Skip to content
academia.sh
tr
Log in

Course Intermediate

Supervised Learning

Published Updated

By the end of this course

Start course

01

Linear Models

Measuring by prediction performance the families that build the decision as a line or a plane: the assumptions checked with the least squares number on the held-out set and the structure of the residuals, the divergence of the train and held-out numbers as the polynomial degree is swept from one to eight and the set from which the choice was read, how many coefficients the ridge, lasso and elastic net penalties set exactly to zero and what sparsity is worth on the held-out set, the continuity of the probability output independent of the threshold and the decision boundary, and the score paid for a readable coefficient together with what the coefficient does not say.

  1. 01 Linear Regression Measuring least squares as a prediction procedure rather than an explanation tool and establishing the three numbers of the course: on the test set the baseline gives a root error of 6.5324 m³ while the linear model gives 5.3096, the difference is 1.2228 m³ and the number of candidates tried is 1. The mean of the residuals is exactly zero per region on the train set and wanders between -1.0769 and +1.3685 on the test set; the residual spread rises from 3.566 to 7.137 as the prediction grows. When the 127 subsets of the seven columns are tried, the subset chosen on validation falls behind the full model with 5.3157 on test; had it been chosen on test, 5.2931 would have been reported and the difference is 0.0226 m³.
  2. 02 Polynomial Regression Building a nonlinear relation with a linear model by adding columns, and the degree pulling two numbers apart: when the degree is swept from 1 to 8 the number of columns rises from 7 to 168, the train root error falls from 5.0757 to 4.8437 m³, and the validation root error bottoms out at 5.2569 and climbs to 5.6877. The degree read from validation is 4 and gives 5.3775 on test; had it been read from test, degree 5 would have been chosen and 5.2009 reported, a difference of 0.1766 m³. The degree chosen on validation is 0.0679 m³ behind degree 1 on test. Four features produce only 450 discrete combinations; the ceiling model that gives every cell the train mean yields 3.8418 on train and 6.3887 on test, and 33 test subscribers fall into a cell never seen in training.
  3. 03 Regularization Penalties Applying the ridge, lasso and elastic net penalties to the same 87 columns and measuring them by sparsity: the ridge penalty sets not a single coefficient exactly to zero in any of six settings, the lasso penalty zeroes from 42 up to 84, and the elastic net penalty stays between 24 and 83. At lasso 10, 65 of the 87 coefficients are exactly zero and the remaining 22 columns give 5.2909 m³ on test; the ridge penalty reaches the same number by keeping all 87 columns, at 5.2954. The choice read from validation is ridge 3 and it gives 5.2954 on test; had it been read from test, lasso 10 would have been chosen and the difference would have been 0.0046 m³. At lasso 3000 all 87 coefficients are zeroed and the model returns exactly to the baseline, 6.5324; the ridge penalty still stays at 6.5086 even at 100000.
  4. 04 Logistic Regression Squeezing the output of a linear plane into a probability and measuring that the probability is continuous before the threshold: with twelve fixed columns the model gives 0.7937 on the test set and passes the baseline by 0.0357, that is, nine subscribers. When nine candidates are tried, five tie at 0.8373 on validation and the test numbers of those five split between 0.7937 and 0.8016; the setting read from validation gives 0.7937 on test, and the difference is two subscribers. All 252 of the 252 subscribers in the test set receive a different probability; when the threshold is swept from 0.20 to 0.80, a single model gives seven different accuracy numbers between 0.7183 and 0.8016, the number of flagged subscribers falls from 102 to 5, and 27 subscribers change class between 0.40 and 0.60.
  5. 05 Coefficient Interpretation Treating readability as a number rather than a praise: the readable model chosen from eight candidates on the validation set carries twelve coefficients and gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056 with 33 columns, and the price of interpretability is 0.0040, that is, a single subscriber. The twelve coefficients give the order of the generating structure in the fiction correctly. The same coefficients are not fixed: when the per-person consumption column is removed, the coefficient of mean consumption rises from 0.772 to 0.808 and the test number does not change at all; when mean consumption is removed, the coefficient of per-person consumption jumps from 0.071 to 0.442, that is, to six times. The ridge penalty brings the coefficients down from 0.772 to 0.326 while the test number falls by only 0.0079.

02

Nonlinear Models

Counting the difference and the cost that the families which come into play when the decision boundary is curved add: the number that the neighbor count and the distance measure move and the nearest and farthest neighbors drawing closer as columns increase, the probability output running to the extremes when the conditional independence assumption breaks while the ranking survives, the margin width and the number of support vectors that carry the decision and the result the kernel transformation gives without being expanded, the split a splitting criterion chooses and how pruning differs from a depth limit, the number that settles and the variability that falls as the number of trees grows together with the out-of-bag estimate, the trade-off between learning rate and number of rounds in sequential boosting, the difference three combinations add and the number of examples on which the ensembles err together.

  1. 01 Nearest Neighbor Measuring a family that computes nothing and stores the training set as it is: twenty-four candidates are tried over the neighbor count and the distance metric, the candidate chosen on the validation set gives 0.7937 on the test set and beats the baseline by 0.0357, yet the same candidate scores 0.8294 on the set it was chosen on. A procedure that looks at one neighbor falls below the baseline on all three metrics. As the column count grows, the ratio of the farthest neighbor to the nearest drops from 26.91 to 1.49 and the model settles exactly on the baseline.
  2. 02 Probabilistic Classifier with an Independence Assumption Measuring a family that counts columns instead of summing them: instead of the 201,600 combinations of eight columns, a count table of 80 cells is kept, twenty candidates are tried and the candidate chosen on the validation set gives 0.7817 on the test set. A zero count eliminates a class entirely; with 60 training rows, 137 test rows are vetoed, and smoothing removes the veto but does not guarantee accuracy. When the column carrying the same information is added eight times, subscribers with extreme probabilities rise from 2 to 169 and the mean confidence from 0.842 to 0.970; the hits in the top 61 of the ranking go from 29 to 33.
  3. 03 Margin-Based Classifier Measuring a family that bases its decision only on the examples near the boundary: as the penalty coefficient is swept, the margin widens from 0.047 to 93.488, the candidate chosen on the validation set gives 0.7857 on the test set and the rule that resolves the tie alone is worth four subscribers. The decision is carried by the 324 rows inside the margin; dropping 300 rows from outside the margin changes 4 predictions, dropping 300 rows from inside changes 21. Nine candidates are tried with three kernels and the curved boundary cannot beat the straight one; the polynomial kernel gives the same prediction on 252 of 252 test rows as 90 explicitly computed columns, without opening those 90 columns at all.
  4. 04 Decision Trees Measuring the split chosen by the split criterion and how pruning differs from a depth limit: impurity and entropy choose the same root split but agree on only eighteen of forty-seven splits, while the misclassification rate chooses an entirely different 695/61 split. The unpruned depth-12 tree with 127 leaves gives 0.7341 on the test set, below the baseline; pruning brings the same tree down to 20 leaves and up to 0.7857. The depth limit gives 0.7897 with 48 leaves while pruning gives 0.7857 with 11 leaves, and four of those are deeper than the sixth level. The trees of eight 90% subsamples share not a single split.
  5. 05 Bagged Tree Ensembles Counting the fragility of a single tree and measuring what bagging adds: across 41 bag samples the same procedure chooses nine different column-threshold pairs for the root split, two trees classify on average 46.2 of the 252 subscribers in the validation set differently, and the single tree's test number wanders between 0.6905 and 0.7897. As the number of trees goes from 1 to 41, the spread on the validation set falls from 0.0193 to 0.0075. When feature randomness is added, the ensemble chosen on the validation set among 31 candidates gives 0.7897 on the test set; the out-of-bag estimate, 0.7884, stands 0.0013 from this number, while the validation set's own number stands 0.0516 away.
  6. 06 Sequential Boosting Measuring what happens when trees are built on the remaining error instead of independently of one another: over two hundred rounds the training number climbs from 0.7778 to 0.8638, while the validation number peaks at 0.8373 in round ten and drops to 0.8056. The learning rate and the number of rounds substitute for each other; the same target is reached in 79 rounds with a rate of 0.02, in 30 with 0.05, in 10 with 0.15 and in 4 with 0.45, and the product of rate and rounds stays between 1.50 and 1.80. On the seventy-two-cell grid, the candidate chosen from validation gives 0.7738 on the test set and the one chosen from the test set gives 0.8056: a difference of 0.0318. In total 1,279 candidates were tried, and boosting with 30 splits fell 0.0159 behind bagging with 568 splits.
  7. 07 Ensemble Methods Counting side by side what bagging, boosting and stacking add and what they cost: the three base models give 0.7778, 0.7897 and 0.7738 on the test set at costs of 43, 568 and 30 splits. On 45 of the 252 subscribers all three are wrong, bagging and boosting classify only 6 subscribers differently, and no combination that sees nothing but the three labels can pass 0.7897. When the second layer of stacking is learned from the validation set it reports 0.8254 and gives 0.7778; when it is learned from the training set it reports 0.8532 and again gives 0.7778: the inflation goes from 0.0476 to 0.0754 and the weight shifts to the model that memorizes most. The topic's total tuning budget is 1,313 candidates.

03

Evaluation and Tuning

Separating out which measure, which threshold and how many candidates a model's number came from: the different error distribution of two models giving the same accuracy and the different cost of the two error types, the ranking changing across accuracy, precision, recall and F1, the opposite movement of precision and recall as the threshold is swept and which curve is misleading on an imbalanced set, the difference between the stated probability and the realized rate and the existence of a model whose ranking is good but whose calibration is bad, mean absolute error, mean squared error and the explained variation ratio not giving the same ranking, the estimate shifting as the number of folds grows and the number of trainings rising, the best candidate three searches find with the same budget and the validation number that inflates as the budget grows, the slice where the error concentrates and the number of bad slices that grows by chance as the slices multiply.

  1. 01 Confusion Matrix Separating the error distributions of two models with the same accuracy: among forty-nine candidates, fourteen accuracy values are shared by more than one candidate, two candidates giving 0.7817 catch 16 of the 61 suspicious subscribers in one and 32 in the other, and the number of flagged subscribers ranges from 26 to 58. When a field visit counts as 1 and a missed illicit case as 6, the cost is 296 and 232, the baseline costs 366, and calling everyone suspicious at 252 is cheaper than the baseline. The best candidate by accuracy gives 263 in cost, while the best candidate by cost is twentieth among forty-nine candidates in accuracy.
  2. 02 Classification Metrics Ranking five models by four metrics derived from the confusion matrix: the model that is first in accuracy is second in precision and fourth in recall and the F1 score, three different models take the lead on the four metrics, and seven of the ten model pairs swap places between accuracy and F1. Because the baseline flags no subscriber, its precision is undefined and its recall and F1 score are zero. For a procedure with precision 1.0000 and recall 0.0164, the arithmetic mean is 0.5082 while the F1 score is 0.0323, and the difference rises to 0.4759.
  3. 03 Threshold Selection and Curves Separating the two numbers the threshold moves from the area under the curve: as the threshold goes from 0.10 to 0.90, precision rises from 0.3608 to 1.0000, recall falls from 0.9344 to 0.0328 and the cost rises from 182 to 356. The 0.16 chosen with the validation set among ninety-nine threshold candidates gives a cost of 181 on the test set, is 86 units cheaper than the default threshold of 0.50, and would have looked 7 units better if it had been read from the test set. When the suspicious share is lowered from 0.2421 to 0.0354, the area under the ROC curve rises from 0.8058 to 0.8624 while that of the precision–recall curve falls from 0.6212 to 0.4124.
  4. 04 Probability Calibration Measuring the difference between the stated probability and the realized share: the calibration error of the linear model is 0.0242, that of the sharpened score, whose ranking is exactly equal to it, is 0.1512, and the area under the curve of the two models is also 0.8058. The calibration error of the depth 8 tree is 0.0000 on the training set by definition and 0.1435 on the test set; the calibration mapping learned from training converges to the identity mapping and corrects nothing. The single-variable mapping learned from the validation set lowers the error of the sharpened score from 0.1512 to 0.0463 and leaves the area under the curve at 0.8058.
  5. 05 Regression Metrics The shift of the measure from counting to distance when the target is a number, and two members of the same family trading places on three measures: the linear model fitted to squared error leads on mean squared error with 22.3164, the one fitted to absolute error leads on mean absolute error with 3.6178. The baseline is the train mean (19.7665 m³) and the coefficient of determination is defined exactly relative to this baseline. 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; when those 13 rows are trimmed, the ranking reverses on both measures. The number of candidates tried is three and no hyperparameter was swept.
  6. 06 Cross-Validation Measuring the number of folds itself: when k is swept from 2 to 10, the mean of the estimate stays between 0.8115 and 0.8204, the true number on the test set is 0.8056 and the closest estimate to the truth comes from k=4. Leave-one-out gives 0.8175, 0.0119 away from the truth, but for this it processes 1,008 trainings and 1,015,056 rows — 111.89 times k=10. Its per-fold spread is 0.3863 and this number is not uncertainty but the correct/incorrect scatter of a one-row fold; because none of the 1,008 folds contains both classes, precision and recall cannot be computed per fold. On the other hand, the dependence of leave-one-out on the shuffle is exactly zero.
  7. 07 Hyperparameter Search The lesson where the rule of the course is paid: with the same budget of 50 candidates, the best candidate found by grid, random and model-based search and the number of that candidate on the test set are printed side by side. As the number of candidates rises from 1 to 200, the best number on the validation set climbs from 0.7937 to 0.8452, the number of the same candidate on the test set goes from 0.7778 to 0.7976, and the inflation between them opens from +0.0159 to +0.0476. The untuned default candidate already gives 0.8452 on the validation set, that is, 200 candidates found nothing. At budgets of 50 and 200 the single-split decision rule beats the linear model on the test set: the family difference is −0.0159 and −0.0040 while the inflation by the budget is +0.0516 and +0.0476.
  8. 08 Error Analysis Breaking the 51 errors on the test set down by region, tariff step, household size and number of readings: in two of the fourteen measurable slices the model falls below the baseline, and the central region holds 23% of the rows while carrying 35.3% of the errors. As the number of slices goes from 14 to 41 and from 41 to 50, the number of bad slices found goes from 2 to 6 and from 6 to 9 — but when the same errors are distributed randomly over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, that is, the observed number is below chance in all three sets. The course wrap-up gathers the family, the number of candidates and the margin above the baseline of twenty lessons in a single table.

Final exam

Supervised Learning — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close