Unit 01 of 03
Linear Models
Measuring by prediction performance the families that build the decision as a line or a plane: the assumptions checked with the least squares number on the held-out set and the structure of the residuals, the divergence of the train and held-out numbers as the polynomial degree is swept from one to eight and the set from which the choice was read, how many coefficients the ridge, lasso and elastic net penalties set exactly to zero and what sparsity is worth on the held-out set, the continuity of the probability output independent of the threshold and the decision boundary, and the score paid for a readable coefficient together with what the coefficient does not say.
Published Updated
Lessons
- 01 Linear Regression Measuring least squares as a prediction procedure rather than an explanation tool and establishing the three numbers of the course: on the test set the baseline gives a root error of 6.5324 m³ while the linear model gives 5.3096, the difference is 1.2228 m³ and the number of candidates tried is 1. The mean of the residuals is exactly zero per region on the train set and wanders between -1.0769 and +1.3685 on the test set; the residual spread rises from 3.566 to 7.137 as the prediction grows. When the 127 subsets of the seven columns are tried, the subset chosen on validation falls behind the full model with 5.3157 on test; had it been chosen on test, 5.2931 would have been reported and the difference is 0.0226 m³. 15 min read
- 02 Polynomial Regression Building a nonlinear relation with a linear model by adding columns, and the degree pulling two numbers apart: when the degree is swept from 1 to 8 the number of columns rises from 7 to 168, the train root error falls from 5.0757 to 4.8437 m³, and the validation root error bottoms out at 5.2569 and climbs to 5.6877. The degree read from validation is 4 and gives 5.3775 on test; had it been read from test, degree 5 would have been chosen and 5.2009 reported, a difference of 0.1766 m³. The degree chosen on validation is 0.0679 m³ behind degree 1 on test. Four features produce only 450 discrete combinations; the ceiling model that gives every cell the train mean yields 3.8418 on train and 6.3887 on test, and 33 test subscribers fall into a cell never seen in training. 5 min read
- 03 Regularization Penalties Applying the ridge, lasso and elastic net penalties to the same 87 columns and measuring them by sparsity: the ridge penalty sets not a single coefficient exactly to zero in any of six settings, the lasso penalty zeroes from 42 up to 84, and the elastic net penalty stays between 24 and 83. At lasso 10, 65 of the 87 coefficients are exactly zero and the remaining 22 columns give 5.2909 m³ on test; the ridge penalty reaches the same number by keeping all 87 columns, at 5.2954. The choice read from validation is ridge 3 and it gives 5.2954 on test; had it been read from test, lasso 10 would have been chosen and the difference would have been 0.0046 m³. At lasso 3000 all 87 coefficients are zeroed and the model returns exactly to the baseline, 6.5324; the ridge penalty still stays at 6.5086 even at 100000. 5 min read
- 04 Logistic Regression Squeezing the output of a linear plane into a probability and measuring that the probability is continuous before the threshold: with twelve fixed columns the model gives 0.7937 on the test set and passes the baseline by 0.0357, that is, nine subscribers. When nine candidates are tried, five tie at 0.8373 on validation and the test numbers of those five split between 0.7937 and 0.8016; the setting read from validation gives 0.7937 on test, and the difference is two subscribers. All 252 of the 252 subscribers in the test set receive a different probability; when the threshold is swept from 0.20 to 0.80, a single model gives seven different accuracy numbers between 0.7183 and 0.8016, the number of flagged subscribers falls from 102 to 5, and 27 subscribers change class between 0.40 and 0.60. 9 min read
- 05 Coefficient Interpretation Treating readability as a number rather than a praise: the readable model chosen from eight candidates on the validation set carries twelve coefficients and gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056 with 33 columns, and the price of interpretability is 0.0040, that is, a single subscriber. The twelve coefficients give the order of the generating structure in the fiction correctly. The same coefficients are not fixed: when the per-person consumption column is removed, the coefficient of mean consumption rises from 0.772 to 0.808 and the test number does not change at all; when mean consumption is removed, the coefficient of per-person consumption jumps from 0.071 to 0.442, that is, to six times. The ridge penalty brings the coefficients down from 0.772 to 0.326 while the test number falls by only 0.0079. 13 min read