Unit 01 of 02

Clustering

Counting how far a result of unlabeled grouping of points departs from a random partition: three entirely different partitions that three different similarity definitions draw from the same dataset, the different results that different starting points produce at the same number of clusters, and the uncertainty of the elbow, the number of points that single, complete and average linkage separate and the cut height being a decision, the number of clusters and noise points that the neighborhood radius and the minimum point count move, the uncertainty carried by soft assignment disappearing in hard assignment, three internal validity measures choosing three different numbers of clusters, and finding clusters even in a dataset with no structure.

Published October 7, 2026 Updated October 21, 2026

Lessons

01 The Clustering Problem Building the measure itself once the label is removed: the same 756 subscribers are split into four clusters with three different similarity definitions and three entirely different partitions come out — the partitions disagree on a share of point pairs between 0.3750 and 0.4801. Three baselines are defined; under the standard score definition a single cluster gives 6.0000, random assignment 5.9790, slicing by a single column 4.9044 and the method 3.8910, so the method's difference is 2.0880 or 1.0134 depending on which baseline is written. The numbers of the three definitions cannot be compared with each other; the proportional gains are 0.525, 0.349 and 0.295. 12 min read
02 k-Means Measuring iteration and the number of clusters separately: thirty random starts at the same k produce 26 distinct partitions, the numbers stay between 3.5649 and 3.7763 but the best and the worst run disagree on 0.2844 of the point pairs — even two runs that give the same number to four decimals disagree on 0.0037 of them. The number of clusters is a hyperparameter; in a sweep of 120 runs the random baseline falls from 6.0000 to 5.9181 while the method falls from 6.0000 to 2.2570, and three separate readings of the elbow choose three separate values of k: 3, 4 and 5. 14 min read
03 Hierarchical Clustering A family that turns the number of clusters from an input into a cut decision, and the price of that decision: on the same 200 subscribers, single, complete and average linkage start from the same first merge (0.079) and end at 2.245, 8.917 and 4.994; cut at a height of 2.0 they give 3, 56 and 37 clusters respectively. At four clusters, single linkage beats the random assignment baseline by only 0.1492 with 6.0849 and produces [1, 1, 1, 197], while complete linkage beats it by 1.6922 with 4.5419; the two linkages place 86 of the 200 subscribers in different clusters. 9 min read
04 Probabilistic Clustering Measuring a mixture model in which a point is attached to every cluster with a responsibility: the fit score of the random assignment baseline is -4.2424, that of the model found by iterative estimation is -3.7755, a difference of +0.4669. The same difference comes out as +0.3358 on the dataset whose columns are shuffled within themselves, so only +0.1311 of the difference comes from the data. For 74 of the 756 points the largest share is below 0.60, and for these points the mean largest share is 0.5418. On moving to hard assignment the mean uncertainty falls from 0.3797 to zero and the cluster sizes deviate by up to 28.8 points from the soft mass. At a second start, 43.24% of the undecided points and 7.20% of the certain points change cluster. 15 min read
05 Clustering Evaluation Measuring the measures that choose the number of clusters: on the same 756 subscribers the silhouette coefficient chooses 3 clusters, the scatter ratio 6 and the gap measure 10, and none of them is more correct than another. On a random cloud that carries no structure the same three measures still choose 9, 7 and 9; the highest silhouette is 0.2142 on the real data and 0.1255 on the structureless cloud. The label is opened for the first and only time in the course, and the pair agreement rate comes out as 0.5235 for the best partition: the single cluster baseline, at 0.6539, beats all nine partitions. Twenty-seven comparisons were made, and none of the internal measures chose the number of clusters that the label points to. 15 min read