Unit 02 of 02
Recommender Systems
Measuring recommendation in a setup where an item that is not shown is not scored and an item that is not scored is not shown: how surprisingly high recommending the most popular item to everyone is in a sparse subscriber-item matrix and the difference between implicit and explicit feedback, the number of subscribers for whom content-based recommendation cannot produce any recommendation and how much the recommendations resemble one another, the ranking that moves with the number of neighbors and the subscribers left without neighbors as sparsity grows, the two separate rankings that come from counting an unobserved cell as zero versus skipping it, how many subscribers and items are cold and how many each remedy warms up and at what cost, a ranking that looks good offline giving a different number when it collects data with its own decisions, and the number of items that are never shown and the growth of popularity bias when the system is fed with its own recommendations.
Published Updated
Lessons
- 01 The Recommendation Problem Measuring recommendation in a fictional subscriber-item interaction matrix: 360 subscribers and 24 service items meet in only 577 of 8,640 cells, the sparsity is 0.9332, and 80 subscribers have no interaction at all. The baseline is to offer everyone the item chosen most often in training; its hit rate at five is 0.6527, against 0.2036 for a random order. When the same data is read implicitly and read with explicit scores, the two top fives share only one item, recommending by average score gives a hit rate of 0.1497, 16 of the 24 items have fewer than ten scores, and the 5,638 cells that were never shown have a score count of zero. 13 min read
- 02 Content-Based Recommendation Measuring a recommendation that builds a subscriber profile from the features of the item: with an eleven-component binary feature vector a profile is built for 280 of the 360 subscribers, and no recommendation can be produced at all for 80. The hit rate at five is 0.4431, below the 0.6527 of the popularity baseline, but the catalog coverage is 24 instead of 9. The price is similarity: the intra-list similarity of the top-five lists is 0.6769 and 0.715 of the items in a list come from a single domain. A new item that has never been shown is ranked for all 280 subscribers, enters the top ten of 191 of them, and has an average position of 11.55 among 25 items; the same item is last in the popularity ranking. 13 min read
- 03 Collaborative Filtering Measuring recommendation that is produced not from the features of the item but from subscribers resembling one another, and where this similarity breaks in sparsity: in a matrix with 410 training interactions the baseline of the most chosen item gives 0.6527 hit rate@5 and shows 9 of the 24 items, while user-based filtering with three neighbors gives 0.6946 and shows 16 items, but 4.07 of its top five already belong to the baseline list. The number of neighbors is a hyperparameter and moves between 0.5988 and 0.6946 across seven candidates; the item-based setup stays at 0.6467 at its best setting and falls below the baseline. When the training sparsity is raised from 0.9525 to 0.9888, the number of held-out subscribers with no neighbor at all goes from 0 to 105, and the measure staying in place comes not from the method but from the order of the catalog. 15 min read
- 04 Matrix Factorization Measuring the difference between counting an unobserved cell of an interaction matrix with missing entries as zero and leaving it out of the computation altogether: the same routine, with a one-line change, produces two separate rankings from the same 410 training interactions. The skipping setup gives 0.7006 hit rate@5 with four factors and beats the 0.6527 baseline, but shows only 9 of the 24 items; the zero-counting setup falls to 0.2395 with eight factors while showing 20 items, and lowers the share of the top-five lists falling on the five most shown items from 0.7868 to 0.4467. The number of factors is a hyperparameter, six candidates were tried, and the number of items shared by the top fives of the two setups falls from 4.02 to 0.86. 14 min read
- 05 Cold Start Building the recommendation for a subscriber with no interactions and an item never selected: of 360 subscribers 80 have none and 113 have a single interaction, and once the single interaction is hidden 193 subscribers count as cold; in the training set of 464 interactions one item was never selected and eight items are below five. The popularity fallback warms all 193 subscribers but gives all of them the same five items, the feature bridge brings the one cold item to 114 subscribers but warms no cold subscriber, and the three-question intake warms 71 subscribers with 339 questions and raises coverage from 5 to 13 items while lowering the top-five hit rate from 0.5752 to 0.5487. 13 min read
- 06 Recommendation Evaluation Building ranking measures and showing that the same ranking gives two different numbers: hit rate at k reorders three setups depending on the cutoff, the setup that is last at k=3 is first at k=5 and at k=10 all three are indistinguishable, and positional discount drops the first place of k=5 to last. The setup that is first on the offline log with 0.6886 falls last, below the baseline, with 1.4563 interactions per subscriber once it starts collecting data with its own decisions; 0.6305 of the held-out cells were never shown, 0.5868 of the held-out items come from five items, and three items never appear in the held-out set. 8 min read
- 07 Diversity and the Echo Chamber The narrowing of coverage and the growth of popularity bias when the system is fed by its own recommendations: over forty-eight periods the narrow setup after the eighth period never shows 8 of 24 items in any period, six items appear in none of the last twenty-four periods, and the impression share of the top five items rises from 0.4711 to 0.5461. The setup that gives the fifth slot to diversity loses 1.0000 interactions per subscriber in the first eleven periods, finds the item whose conditions changed in the twelfth period and moves ahead, breaks even cumulatively in the twenty-seventh period and finishes the forty-eighth period 1.2250 ahead. 13 min read