Unit 02 of 02
Applied Statistics
Testing with a permutation test whether a difference is real or noise and reporting the result with its margin of variation: the decision between mean and median moving the reported number, what box and density summaries squeeze into five numbers and what they hide, what a confidence interval says and does not say, test choice and assumption checking, the number of decisions an experiment set yields through the garden of forking paths, the interpretive limit of a regression coefficient and how much of the effect it explains, the retention rate moved by the cohort definition, selection bias, and counting the difference found by trying many things.
Published Updated
Lessons
- 01 Choosing Descriptive Summaries Three defensible summaries come out of the same 557 readings: the mean is 24.674 m³, the median 24.120 m³ and the 10% trimmed mean 24.581 m³, with a largest difference of 0.554 m³ among them. Bootstrap measures the margin of variation of all three and the widths come out as 1.552, 1.762 and 1.458 m³: the difference the choice moves is a third of the margins, and all three intervals contain the 25 m³ tariff step. When the same computation drops from the 557-reading group to the 142-reading group, the choice difference rises to 1.679 m³ and the margin width to 2.917 m³, and this time the decision cannot be read independently of the margin. 13 min read
- 02 Distribution Comparison The 921 readings of the north region come down to five numbers, and a second dataset that gives the same five numbers is built: both box summaries are 0.00 | 15.74 | 19.66 | 24.63 | 45.47, the interquartile range is 8.89 m³ and the fences are 2.405 and 37.965 m³. Bin counts separate the two — in north a single peak in the 16–20 m³ bin, in the twin two peaks in the 12–16 and 24–28 m³ bins with the median left in the trough. The notch of the same median comes out as 0.920 m³ and 3.146 m³ with bootstrap, and the number of observations outside the fences varies between 50 and 35. 8 min read
- 03 Interpreting Confidence Intervals The east region's mean of 24.674 m³ settles into a 23.894–25.431 m³ interval with bootstrap and the width comes out as 1.537 m³, but only 37 of the 557 readings are inside this interval: the interval belongs to the summary, not to the observations. When the observation count rises from 35 to 139 and from 139 to 557, the width falls from 6.601 to 3.009 and then to 1.459 m³. In 113 of 120 samples of forty observations the interval contains the population mean. The two periods of north give two intervals that overlap by 0.148 m³, while the interval of the difference itself, 0.281–2.612 m³, leaves zero outside. 4 min read
- 04 Applying Hypothesis Testing Among the 1,101 subscribers read in both periods the observed difference is 0.957 m³, and this single difference gives two separate p-values: in the independent assumption 24 of 2,000 permutations exceed the difference and p comes out as 0.01249, in the paired assumption 5 exceed it and p becomes 0.00300. The assumption check shows with a correlation of 0.213 that the pairing is real; when the pairing is broken p returns to 0.01099. The effect sizes, 0.112 and 0.089, stand next to p. In a quarter of the same dataset, 275 pairs, the difference is larger at 1.132 m³, yet p comes out thirty-five times larger at 0.10695. 13 min read
- 05 A/B Test Analysis Counting the four analytic degrees of freedom in front of the analyst once the dataset of a finished-design experiment is in hand: the choice of metric, the breakdown, the observation window and the handling of the unbalanced assignment produce 108 combinations. The single pre-registered decision gives a 0.123 m³ difference, a margin of variation of [-0.698, 0.952] and a p-value of 0.786; of the 108 combinations walked on the same dataset, 4 fall below the 0.05 threshold, 9 below 0.10, 45 point toward an increase and 63 toward a decrease, and the difference in the mean metric wanders between -2.113 and 1.119 m³. The true difference between the groups is exactly zero. Pre-registration brings the number of combinations from 108 down to 1. 11 min read
- 06 Modeling Relationships with Regression Least squares is written inside the lesson, and what a coefficient says is separated from what it does not say: the univariate coefficient of meter age comes out as -0.605 m³/year, the margin of variation [-0.714, -0.503] and the coefficient of determination 0.098. When the base level of the region is added as a confounder, the same coefficient turns to 1.825, its margin becomes [1.629, 2.003] and the two intervals never overlap. In each of the five regions the within-group slope is positive (between 0.771 and 2.416). The coefficient of determination rises from 0.098 to 0.505 but does not announce the sign; when a completely random column is added it rises again, from 0.505363 to 0.505377. The regional mean of the residuals wanders between -3.020 and 2.481 in the univariate model and drops to between -0.229 and 0.295 when the confounder is added. 13 min read
- 07 Cohort and Funnel Analysis Counting how many separate retention rates the cohort definition produces from the same dataset: the rule that counts the unobservable cohort as zero gives 0.619, the rule that excludes it gives 0.867, and outside the three-period window 843 of the 3,199 readings remain. Taking the entry window as a period gives two observable cohorts and a rate range of 0.009; taking it as five weeks gives six cohorts and a range of 0.040; since the narrowest margin of variation is 0.087, the whole range fits inside the margin of a single cohort. The funnel enters only as far as the aggregation reversal: while the aggregated difference is -0.237, the difference in both of the two observable cohorts is +0.083 and +0.028. 11 min read
- 08 Misleading Analyses Counting the fact that the dataset itself is a choice and that noise can produce findings: the east-north difference is 4.659 m³ in the population of all meters in the network, 4.463 m³ in the sample of readable meters, 4.655 m³ when zero readings are dropped and 4.454 m³ among subscribers who stayed only three periods; the range of the four decisions is 0.206 m³. Of the 1,329 subscribers, 69 appear in no table. Ten scans of twenty breakdowns each run 200 tests on a completely random outcome variable and give 11 findings; the expected number is 10.0 and at least one finding appears in 7 of the ten scans. In the course wrap-up the observed number, margin of variation and whether the decision changes for the thirteen lessons are gathered in a single table. 8 min read