Content-Based Recommendation

The baseline of the previous lesson raised the hit rate at five to 0.6527, but it did so by showing only 9 of the catalog’s 24 items, and to the 80 subscribers with no interaction it said the same thing it said to everyone. Both shortcomings come from the same place: the baseline looks only at the interaction counts; it does not look at what the item is.

The catalog has written information per item: which domain it belongs to, which channel it runs through, whether it is paid, and whether it requires a commitment and an appointment. This lesson builds a similarity from those columns, describes the subscriber as the average of the items they selected, and measures two numbers: for how many subscribers no recommendation can be produced at all and how similar the recommendations produced are to one another.

• RS9. The setup and the seed are the same as in the previous lesson; the matrix, the split and the popularity baseline are rebuilt exactly and give the same numbers.
• RS10. The feature vector has eleven components: five domains, three channels and three binary flags. The components are 0/1 and the vector is brought to unit length.
• RS11. The subscriber profile is the sum of the feature vectors of the items the subscriber selected in the training set, brought to unit length. Scores are not used; the profile comes from the implicit signal.
• RS12. The similarity of two unit vectors is their dot product and lies between 0 and 1. In case of a tie the catalog order breaks it; floating-point values are not compared directly but used as a sort key.
• RS13. Coverage is two separate numbers: for how many subscribers a recommendation could be produced and how many distinct items entered the list of at least one subscriber.
• RS14. The intra-list similarity of a list is the average similarity of all item pairs in the list.
• RS15. At the end there is a new item in the catalog: its features are written, it has never been shown, and it has no interaction.

Rebuilding the Matrix

If the measurement is not done on the same data, the numbers of the two lessons cannot be compared. The block below rebuilds the setup, the split and the popularity baseline with the same seed; its two-line output must be exactly the same as the sparsity and hit rate numbers of the previous lesson.

# COMMON -- FICTIONAL. The matrix of the previous lesson is rebuilt with the same seed; this time
# the feature columns of the catalog will be read too.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


ITEMS = [  # (name, domain, channel, paid, commitment, appointment, base_share)
    ("auto_payment_instruction", "billing", "online", 0, 0, 0, 0.90),
    ("e_invoice_switch", "alert", "online", 0, 0, 0, 0.85),
    ("payment_reminder_sms", "alert", "online", 0, 0, 0, 0.78),
    ("installment_plan", "billing", "branch", 0, 1, 0, 0.62),
    ("remote_reading_application", "meter", "field", 0, 1, 1, 0.55),
    ("outage_alert_subscription", "alert", "online", 0, 0, 0, 0.50),
    ("debt_restructuring", "billing", "branch", 0, 1, 0, 0.44),
    ("address_update", "subscription", "online", 0, 0, 0, 0.40),
    ("meter_replacement", "meter", "field", 1, 0, 1, 0.36),
    ("consumption_report_subscription", "alert", "online", 0, 0, 0, 0.32),
    ("invoice_dispute_record", "billing", "branch", 0, 0, 0, 0.28),
    ("subscription_transfer", "subscription", "branch", 0, 0, 0, 0.25),
    ("illicit_use_check_appointment", "meter", "field", 0, 0, 1, 0.22),
    ("installation_maintenance_package", "technical", "field", 1, 1, 1, 0.19),
    ("second_meter_request", "meter", "field", 1, 0, 1, 0.16),
    ("business_tariff_switch", "subscription", "branch", 0, 1, 0, 0.14),
    ("pressure_regulator_mounting", "technical", "field", 1, 0, 1, 0.12),
    ("garden_irrigation_tariff", "subscription", "branch", 0, 1, 0, 0.10),
    ("meter_relocation", "meter", "field", 1, 0, 1, 0.09),
    ("common_area_sharing_record", "subscription", "branch", 0, 1, 0, 0.08),
    ("installation_compliance_report", "technical", "field", 1, 0, 1, 0.07),
    ("late_interest_waiver", "billing", "branch", 0, 0, 0, 0.06),
    ("mobile_access_permission", "alert", "online", 0, 0, 0, 0.05),
    ("extra_line_installation", "technical", "field", 1, 1, 1, 0.04),
]
SEGMENT = {
    "saver":    {"billing": 0.5, "meter": 1.7, "subscription": 0.6, "alert": 1.5, "technical": 0.6},
    "payer":    {"billing": 1.9, "meter": 0.5, "subscription": 0.6, "alert": 1.1, "technical": 0.3},
    "business": {"billing": 0.7, "meter": 0.9, "subscription": 1.6, "alert": 0.7, "technical": 1.9},
    "new":      {"billing": 0.8, "meter": 0.7, "subscription": 1.8, "alert": 1.3, "technical": 0.7},
}
SEGMENT_NAMES = list(SEGMENT)
rq = rng(SEED + 101)
QUALITY = [0.25 + 0.70 * rq() for _ in ITEMS]

N_SUBSCRIBERS, N_ITEMS = 360, len(ITEMS)
SUBSCRIBERS, INTERACTIONS = [], {}
for i in range(N_SUBSCRIBERS):
    r = rng(SEED + 7919 * i)
    a, segment = 20001 + i, SEGMENT_NAMES[int(r() * 4)]
    activity = 0.35 + 1.15 * r() ** 1.6
    SUBSCRIBERS.append(a)
    for j, (name, domain, channel, paid, commit, appt, share) in enumerate(ITEMS):
        propensity = min(0.95, SEGMENT[segment][domain] * share * activity * 0.55)
        if r() < min(0.95, share ** 0.55 * 0.72):     # was it shown
            if r() < propensity:                       # was it selected
                INTERACTIONS[(a, j)] = r()
                if r() < 0.32:
                    r()                                # score draw: not read here
        else:
            r(), r(), r()

TRAIN, HELD_OUT, SEEN, log = {}, {}, {}, {}
for (a, j), stamp in INTERACTIONS.items():
    log.setdefault(a, []).append((stamp, j))
for a, entries in log.items():
    entries.sort()
    if len(entries) >= 2:
        HELD_OUT[a] = entries[-1][1]
        entries = entries[:-1]
    for stamp, j in entries:
        TRAIN[(a, j)] = 1
        SEEN.setdefault(a, set()).add(j)
n_chosen = [0] * N_ITEMS
for (a, j) in TRAIN:
    n_chosen[j] += 1
ranked = sorted(range(N_ITEMS), key=lambda j: -n_chosen[j])


def hit_rate(recommend, K):
    return sum(1 for a in HELD_OUT if HELD_OUT[a] in recommend(a)[:K]) / len(HELD_OUT)


def baseline(a):
    return [j for j in ranked if j not in SEEN.get(a, set())]


print("interaction", len(INTERACTIONS), " sparsity",
      round(1 - len(INTERACTIONS) / (N_SUBSCRIBERS * N_ITEMS), 4))
print("held-out subscribers", len(HELD_OUT), " baseline top 5 hit rate", round(hit_rate(baseline, 5), 4))

interaction 577  sparsity 0.9332
held-out subscribers 167  baseline top 5 hit rate 0.6527

The two lines are exactly the same as the previous lesson: 577 interactions, 0.9332 sparsity, 167 held-out subscribers and a baseline hit rate of 0.6527. This repetition is not a ceremony but a precondition of the measurement: the numbers of two methods are comparable only when they are read on the same split, the same held-out set and the same baseline. Because the setup is regenerated from the same seed, every difference in between comes from the method.

From Feature to Profile

Every item turns into an eleven-component vector and is brought to unit length; this way an item with many features does not stand out for that reason alone. A paid, commitment-bound, appointment-bound field item carries 1 in six components, a free online item in only two; without the scaling the first would look closer to every profile. Because the components are binary, many equal similarities come out, and ties are broken by the catalog order; this is an arbitrary rule and was written to keep the result constant from run to run.

The profile of a subscriber is the sum of the vectors of the items they selected: the profile of a subscriber who made two field applications stretches in the field direction. The recommendation is to rank the items that are closest to the profile and have not been selected before. If the subscriber has no selection at all there is no vector to add up and the profile cannot be built.

import math

DOMAINS = ["billing", "meter", "subscription", "alert", "technical"]
CHANNELS = ["online", "field", "branch"]


def features(k):
    v = [1.0 if k[1] == x else 0.0 for x in DOMAINS]
    v += [1.0 if k[2] == x else 0.0 for x in CHANNELS]
    v += [float(k[3]), float(k[4]), float(k[5])]
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]


def similarity(u, v):
    return sum(x * y for x, y in zip(u, v))


FEAT = [features(k) for k in ITEMS]
DIM = len(FEAT[0])

PROFILE = {}
for a in SUBSCRIBERS:
    seen = SEEN.get(a, set())
    if not seen:
        continue                                   # profile cannot be built
    p = [sum(FEAT[j][d] for j in seen) for d in range(DIM)]
    n = math.sqrt(sum(x * x for x in p))
    PROFILE[a] = [x / n for x in p]


def content_based(a):
    if a not in PROFILE:
        return []                                  # no recommendation can be produced
    return sorted((j for j in range(len(FEAT)) if j not in SEEN.get(a, set())),
                  key=lambda j: (-similarity(PROFILE[a], FEAT[j]), j))


def coverage(recommend, K=5):
    return len({j for a in HELD_OUT for j in recommend(a)[:K]})


def intra_similarity(recommend, K=5):
    t = n = 0
    for a in HELD_OUT:
        entries = recommend(a)[:K]
        for x in range(len(entries)):
            for y in range(x + 1, len(entries)):
                t += similarity(FEAT[entries[x]], FEAT[entries[y]])
                n += 1
    return t / n


print("subscribers with a profile", len(PROFILE), "/", N_SUBSCRIBERS,
      " without", N_SUBSCRIBERS - len(PROFILE))
for K in (3, 5):
    print(f"top {K}: content {hit_rate(content_based, K):.4f}  baseline {hit_rate(baseline, K):.4f}")
print("catalog coverage            content", coverage(content_based), " baseline", coverage(baseline), "/", N_ITEMS)
print(f"top 5 intra-list similarity content {intra_similarity(content_based):.4f}  baseline {intra_similarity(baseline):.4f}")
t = 0
for a in HELD_OUT:
    domains = [ITEMS[j][1] for j in content_based(a)[:5]]
    t += max(domains.count(x) for x in set(domains))
print("mean share of the most frequent domain in the top 5", round(t / len(HELD_OUT) / 5, 4))

subscribers with a profile 280 / 360  without 80
top 3: content 0.2874  baseline 0.5509
top 5: content 0.4431  baseline 0.6527
catalog coverage            content 24  baseline 9 / 24
top 5 intra-list similarity content 0.6769  baseline 0.3827
mean share of the most frequent domain in the top 5 0.715

Content-based recommendation stays below the baseline: its hit rate at five is 0.4431, 0.2096 behind the baseline’s 0.6527; at three the gap is bigger, 0.2874 against 0.5509. The source of the gap is what the method sees. The probability of an item being selected comes in the setup from two factors: the subscriber’s propensity toward that domain and the item’s own prevalence. The feature vector carries only the first; the second, that is how many subscribers selected that item, is not written in the feature columns at all. The method replaces the question “what will this subscriber select” with the question “what resembles what this subscriber selected” and drops the prevalence.

The coverage column looks the other way. The baseline put only 9 items into the lists of 167 subscribers; content-based recommendation puts all 24 items into at least one list. On the other hand the lists narrow within themselves: the intra-list similarity of the top five is 0.3827 for the baseline and 0.6769 for content-based recommendation, and on average 0.715 of the items in the top five come from a single domain. To a subscriber who made a field application, four more field applications are recommended. Diversity increases at the catalog level and decreases at the subscriber level, and if the two numbers are not written separately, coverage alone misleads.

Subscriber coverage is the other half of coverage: a profile can be built for 280 of the 360 subscribers, and no recommendation can be produced at all for 80. These 80 subscribers will see nothing in the next period either, will select nothing, and their rows will keep staying empty.

Three numbers answer three separate questions and none replaces another. The hit rate asks “is the held-out item in the list” and the content-based method loses here. The catalog coverage asks “how many items never appear” and it wins here. The intra-list similarity asks “are the five items in the list repetitions of one another” and it loses here. A report that writes only the hit rate makes the method look needless; a report that writes only the coverage makes it look better than it is.

New Item and New Subscriber

Cold start has two directions and content-based recommendation gives two separate answers to them. Think of an item in the catalog that has never been shown: its interaction count is zero, but its features were written together with the catalog. Think also of a subscriber who has no interaction: there are no features, because features belong to the item, not to the subscriber.

# A new item that has never been shown is added to the catalog: it has no interaction,
# only features.
NEW = ("mobile_meter_reading", "meter", "online", 0, 0, 0, 0.0)
FEAT.append(features(NEW))
NJ = len(FEAT) - 1

top5 = sum(1 for a in PROFILE if NJ in content_based(a)[:5])
top10 = sum(1 for a in PROFILE if NJ in content_based(a)[:10])
position = sum(content_based(a).index(NJ) + 1 for a in PROFILE) / len(PROFILE)
print(f"{NEW[0]}: content-based top 5 -> {top5} subscribers, top 10 -> {top10} subscribers")
print("mean position", round(position, 2), "/", len(FEAT), " position in the baseline ranking",
      len(FEAT), "(interaction count 0)")
print("subscribers without a profile", N_SUBSCRIBERS - len(PROFILE), " content-based recommendation count",
      sum(len(content_based(a)) for a in SUBSCRIBERS if a not in PROFILE),
      " given by the baseline", 5 * (N_SUBSCRIBERS - len(PROFILE)))

mobile_meter_reading: content-based top 5 -> 13 subscribers, top 10 -> 191 subscribers
mean position 11.55 / 25  position in the baseline ranking 25 (interaction count 0)
subscribers without a profile 80  content-based recommendation count 0  given by the baseline 400

The new item is ranked for all 280 subscribers: although it has never been shown, its average position is 11.55 among 25 items, and it enters the top ten of 191 subscribers and the top five of 13. In a ranking that relies on interaction counts the same item is last for everyone, and as long as it stays last it cannot collect interactions because it is not shown. The feature is what breaks this loop: it carries a piece of information about the item that is known before the interaction.

The same help is not there on the subscriber side. For the 80 subscribers without a profile the content-based method produces a total of zero recommendations; the baseline without personalization gives the same subscribers 400 recommendations, all the same five items. The asymmetry comes from the definition of the method: the features of the item are written in the catalog, those of the subscriber are not. The only thing known about a subscriber is their past selections, and about a subscriber with no past selection nothing is known.

This is the collected data of this lesson: 80 subscribers receive no recommendation, and because they cannot receive one they add no row to the interaction matrix of the next period either.

Summary

• The feature vector has eleven components; the subscriber profile is the sum, brought to unit length, of the vectors of the items selected, and it can be built for 280 of the 360 subscribers.
• The hit rate at five of content-based recommendation is 0.4431 and at three 0.2874; the popularity baseline gives 0.6527 and 0.5509 on the same measures. The feature columns do not carry the prevalence of the item.
• The catalog coverage rises from 9 to 24, but the intra-list similarity of the lists rises from 0.3827 to 0.6769 and 0.715 of the items in the top five come from a single domain.
• Coverage is two numbers: no recommendation can be produced for 80 subscribers, and because nothing is shown to these subscribers their rows keep staying empty.
• A new item that has never been shown is ranked for all 280 subscribers, enters the top ten of 191 of them and has an average position of 11.55; in the ranking that relies on interaction counts the same item is last.

Next Step

Content-based recommendation counts two items as close because their feature columns are similar. That is why it can rank the new item from the first day, and that is why it recommends five items from the same domain to a subscriber. One thing was never used in the measurement: the similarity of subscribers to one another. Two subscribers who selected the same three items share something that has no trace in the feature columns; a fourth item that one has selected and the other has not yet selected is a recommendation candidate. The next lesson builds the similarity not from the features of the item but from the rows and columns of the interaction matrix, measures how much the number of neighbors moves the ranking, and counts the subscribers for whom no neighbor can be found at all as sparsity grows.