Unit 02 of 02
Dimensionality Reduction and Anomalies
Measuring what is kept and what is lost when the number of columns is reduced, and how a point that stands alone is flagged: the reconstruction error of the projection that keeps the variability, how much of the pairwise distances survive, and the scaling decision changing the projection; the trade-off between the error that grows as the rank falls and the storage cost; how many of the nearest neighbors remain neighbors in a neighborhood-preserving embedding and the distance between two points being uninterpretable; how many of the flagged anomalies are real and the multivariate anomaly the univariate procedure cannot catch; high-confidence rules coming only from a frequent item and lift correcting this.
Published Updated
Lessons
- 01 Principal Component Analysis Measuring what is kept and what is lost when a twelve-column subscriber table is reduced to fewer numbers: reconstruction with column means gives 3.3411 while reconstruction with two components gives 2.7762, and the difference is only 0.169 of the baseline; the same two components carry 0.341 of the component share, yet the mean of the pairwise distances falls to 0.4965 times the original and only 0.0057 of the 31,626 pairs survive within a 10% margin. In the unscaled set the load of the first component binds to a single column with +0.96 and the share of the first two components rises to 0.863, but the reconstruction error measured in a common unit, 2.8473, stays behind the scaled projection. 11 min read
- 02 Singular Value Decomposition Application Measuring the cost of storing a matrix with a low-rank approximation, in error and in storage cost (in numbers): in a hand-built 6x5 matrix the total error of the rank 2 approximation is 0.6746 and this number equals exactly the root of the squares of the discarded singular values, but at rank 3 the storage cost rises to 36 and exceeds the original 30 numbers. In the 252x12 subscriber table reconstruction with column means gives 3.3411 while the rank 2 approximation gives 2.7464 and stores 554 numbers instead of 3,024; storing two random raw columns with the same budget gives 3.0617, and the advantage of the low-rank approximation is 0.3153 at rank 2 and 0.9663 at rank 8. 13 min read
- 03 Nonlinear Embedding Measuring two-dimensional layouts that aim to preserve neighborhoods against the linear projection: a random layout preserves 0.0390 of the ten nearest neighbors, the linear projection 0.2639, and the neighborhood-targeted embedding 0.5032. The same embedding preserves only 0.0071 of the ten farthest pairs while the linear projection preserves 0.6012; the two methods preserve the opposite of each other. After the mean pairwise distance is equalized, the original distance of pairs standing at the same distance in the embedding varies between 3.47 and 9.46, and in a second run with the same setting the embedded distance of the same pair wanders by a mean relative difference of 0.2906. 14 min read
- 04 Anomaly Detection Measuring flagging on 63 anomalies deliberately planted into the 1,260 rows of the fictional subscriber table: at a 5% rate random flagging gives a precision of 0.0488, flagging by a single column 0.3968, the procedure that looks at each column on its own 0.6190, and the reconstruction error with four components 0.4921. The precision ranking is misleading — the three detectors with the highest precision find none of the 24 multivariate anomalies that go to the extreme in no column, while the one with the lowest precision finds fifteen of them, and when the contamination rate rises to 10% it finds all twenty-four. At a rate of 2% three detectors give a precision of 1.0000 and at the same time catch only 25 of the 63 anomalies. 10 min read
- 05 Association Rules Measuring the 166 rules that pass the support floor in the 1,329 baskets of the fictional subscriber table: all six of the top six rules by confidence name the same frequent item as the consequent and their lifts are between 1.0139 and 1.0489; of the 97 rules with confidence above 0.50, 75 have a lift closer to 1 than 0.05. The baseline is the support that random matching produces — when the item columns are shuffled separately, the support of 0.1279 for delay and installment plan does not exceed 0.0467 in any of sixty shuffles, but the support of a rule with a confidence of 0.9246 stays inside the random range. The course wrap-up gathers the baseline, the method's number and the difference of eleven lessons in a single table. 10 min read