Self-Supervised Learning

The previous lesson counted how many labels the four setups ask for. Three of them paid for labels and the fourth earned them with visits. None of them asked this: can the label be derived from the data. In the readings table there is a structure that no one labeled — the trace each subscriber leaves across the periods. Closing one period’s reading and predicting it from the rest asks for no label, because the answer is already in the table.

This lesson sets up that setup and shows that two separate tasks have two separate baselines.

• FC14. Self-supervised learning: the label is not given from outside, it is derived from the data itself. The procedure works as supervised, but the supervision is provided by the data.
• FC15. Pretext task: a task whose answer is already in the data. The aim is not to solve that task but to learn something while solving it.
• FC16. Representation: the numbers that the pretext task produces and that are carried to the main task as features. In this lesson three numbers are carried.
• FC17. The pretext task has its own baseline, and that baseline produces a representation too. Unless the two representations are compared in the same place, the contribution of the pretext task is not counted as measured.
• FC18. The pretext task predicts a numerical value; this kind of task is called regression. The term is kept apart from the regression used in the Statistical Analysis and Exploratory Data Analysis course to model the relationship between variables: here the aim is not to explain a relationship but to predict an unseen value.
• FC19. The measure of regression is the mean absolute error, and that of classification is accuracy. This lesson uses two measures and chooses the baseline of each separately.

Rebuilding the Fictional Dataset

# setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings
# table, the subscriber summary, the fictional label, the train / held-out split and the tree learner.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]
FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count",
            "satisfaction", "household_size", "highest", "lowest"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_subscribers, _dropped, _series = [], [], {}
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2],
         "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
         "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1}
    (_dropped if r() < 0.046 else _subscribers).append(k)
_valid = {a["subscriber_id"] for a in _subscribers}
for k in _subscribers + _dropped:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp(
            (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100
        if k["subscriber_id"] in _valid:
            _series.setdefault(k["subscriber_id"], []).append(t)
        r()

SUBSCRIBERS = []
for k in _subscribers:
    t = _series.get(k["subscriber_id"])
    if not t:
        continue
    mean = sum(t) / len(t)
    a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t,
         "household_size": k["household_size"], "satisfaction": k["satisfaction"],
         "reading_count": len(t), "mean": mean, "highest": max(t),
         "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0),
         "per_person": mean / max(1, k["household_size"]),
         "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0}
    r = rng(SEED + 31000 + a["subscriber_id"])
    a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45)
                           + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3)
                           + 0.45 * (a["satisfaction"] <= 2) - 1.55
                           + (r() + r() - 1.0) * 1.1) > 0 else 0
    SUBSCRIBERS.append(a)

r = rng(SEED + 90001)
TRAIN, HELD_OUT = [], []
for a in SUBSCRIBERS:
    (TRAIN if r() < 0.70 else HELD_OUT).append(a)
positives = lambda k: sum(a["suspicion"] for a in k)
accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k)
_impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0


def fit(k, depth=3, min_size=40):
    """The greedy decision tree of lesson 01: the split criterion and stopping rule are there."""
    best, choice = _impurity(k) * len(k), None
    if depth and len(k) >= min_size:
        for feat in FEATURES:
            for threshold in sorted({a[feat] for a in k})[1:]:
                left = [a for a in k if a[feat] < threshold]
                right = [a for a in k if a[feat] >= threshold]
                m = _impurity(left) * len(left) + _impurity(right) * len(right)
                if m < best - 1e-9:
                    best, choice = m, (feat, threshold, left, right)
    if choice is None:
        return 1 if 2 * positives(k) > len(k) else 0
    lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size)
    return lt if lt == rt else (choice[0], choice[1], lt, rt)


def apply(d, a):
    while isinstance(d, tuple):
        d = d[2] if a[d[0]] < d[1] else d[3]
    return d


print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, "
      f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious")

fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious

The Pretext Task and Its Own Baseline

The pretext task is this: the last reading of each subscriber is hidden and predicted from the earlier readings, the number of readings and the record information. This task requires no door visit; the answer is already in the table, it is just not shown to the model.

Because the task predicts a numerical value, its baseline must be numerical too. The non-learning procedure here is “telling everyone the overall mean of the set”; this is the counterpart in regression of the most frequent class in classification. Least squares is used on the model side and the normal equations are solved by elimination.

# pretext.py — an unlabeled task: the last reading of each subscriber is hidden and predicted
# from the earlier readings. The answer is already in the table, no label is spent.
import statistics
from collections import Counter


def solve(A, b):
    """Linear system of equations: elimination with partial pivoting."""
    n = len(b)
    M = [A[i][:] + [b[i]] for i in range(n)]
    for i in range(n):
        p = max(range(i, n), key=lambda s: abs(M[s][i]))
        M[i], M[p] = M[p], M[i]
        if abs(M[i][i]) < 1e-12:  # singular column: the coefficient is left at zero
            continue
        for s in range(n):
            if s != i and M[s][i]:
                f = M[s][i] / M[i][i]
                for c in range(i, n + 1):
                    M[s][c] -= f * M[i][c]
    return [M[i][n] / M[i][i] if abs(M[i][i]) > 1e-12 else 0.0 for i in range(n)]


def least_squares(X, y):
    k = len(X[0])
    return solve([[sum(x[i] * x[j] for x in X) for j in range(k)] for i in range(k)],
                 [sum(x[i] * t for x, t in zip(X, y)) for i in range(k)])


OVERALL = statistics.fmean(v for a in TRAIN for v in a["series"])


def inputs(a):
    previous = a["series"][:-1]
    return [1.0, statistics.fmean(previous) if previous else OVERALL,
            float(len(a["series"])), float(a["household_size"]), float(a["satisfaction"])]


target = lambda k: [a["series"][-1] for a in k]
WEIGHTS = least_squares([inputs(a) for a in TRAIN], target(TRAIN))
predict_value = lambda a: sum(c * v for c, v in zip(WEIGHTS, inputs(a)))
error = lambda f, k: statistics.fmean(abs(f(a) - a["series"][-1]) for a in k)
W = [34, 12, 12]
fmt = lambda h: str(h[0]).ljust(W[0]) + str(h[1]).rjust(W[1]) + str(h[2]).rjust(W[2])
print("pretext task: prediction of the last reading, measure mean absolute error (m3)")
print(fmt(["procedure", "train", "held-out"]))
print(fmt(["baseline: always the overall mean", f"{error(lambda a: OVERALL, TRAIN):.4f}",
           f"{error(lambda a: OVERALL, HELD_OUT):.4f}"]))
print(fmt(["least squares", f"{error(predict_value, TRAIN):.4f}",
           f"{error(predict_value, HELD_OUT):.4f}"]))
print("coefficients", [round(v, 4) for v in WEIGHTS])

pretext task: prediction of the last reading, measure mean absolute error (m3)
procedure                                train    held-out
baseline: always the overall mean       5.9664      6.3776
least squares                           5.6979      5.9661
coefficients [18.2859, 0.262, -1.5773, -0.0497, -0.1704]

The pretext task beats the baseline on its own measure: on the held-out set the error falls from 6.3776 m³ to 5.9661 m³, a gain of 0.4115 m³. This gain was obtained with zero labels; the only thing spent is the table itself.

The coefficients tell what the model learned. The coefficient of the mean of the earlier readings is 0.2620: a subscriber’s past consumption partly determines the last reading, but weakly. The coefficient of the number of readings is -1.5773; the last reading of subscribers with all three periods is systematically lower, because in the fictional dataset each period falls below the one before. The coefficients of household size and satisfaction are close to zero.

The Contribution of the Representation to the Main Task

What is carried from the pretext task to the main task is not the prediction itself but the relationship between the prediction and the actual value. Three numbers are carried: the value the model expected, the difference between it and the actual value, and the ratio of that difference to the expected value.

So that the comparison is honest, the main task is this time built with a narrow feature set: only the three columns that come from the subscriber record. This is the only information that can be obtained without touching the readings table; the contribution of the representation is read as the difference added on top of this narrowness.

# representation.py — the three numbers derived by the pretext task are carried to the main task
# as features. The representation was learned with zero labels; the main task uses 883 labels.
for a in SUBSCRIBERS:
    b = predict_value(a)
    a["expected"], a["deviation"] = b, a["series"][-1] - b
    a["relative_deviation"] = (a["series"][-1] - b) / (abs(b) + 1.0)
RECORD = ["household_size", "satisfaction", "reading_count"]
REPRESENTATION = ["expected", "deviation", "relative_deviation"]
HAND = FEATURES[:]
most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0]
BASE_HELD = accuracy(lambda a: most_frequent, HELD_OUT)


def measure(name, feats):
    global FEATURES
    FEATURES = feats
    d = fit(TRAIN)
    inside, outside = accuracy(lambda a: apply(d, a), TRAIN), accuracy(
        lambda a: apply(d, a), HELD_OUT)
    FEATURES = HAND
    print(str(name).ljust(38) + str(len(feats)).rjust(7)
          + f"{inside:.4f}".rjust(12) + f"{outside:.4f}".rjust(12)
          + f"{outside - BASE_HELD:+.4f}".rjust(13))


print("main task: illicit-use suspicion, measure accuracy")
print("feature set".ljust(38) + "count".rjust(7) + "train".rjust(12)
      + "held-out".rjust(12) + "vs baseline".rjust(13))
print("baseline: always most frequent class".ljust(38) + "0".rjust(7)
      + f"{accuracy(lambda a: most_frequent, TRAIN):.4f}".rjust(12)
      + f"{BASE_HELD:.4f}".rjust(12) + "+0.0000".rjust(13))
measure("record only", RECORD)
measure("record + representation", RECORD + REPRESENTATION)
measure("nine hand-derived features", HAND)

main task: illicit-use suspicion, measure accuracy
feature set                             count       train    held-out  vs baseline
baseline: always most frequent class        0      0.7350      0.7533      +0.0000
record only                                 3      0.7939      0.7851      +0.0318
record + representation                     6      0.8313      0.8037      +0.0504
nine hand-derived features                  9      0.8709      0.8594      +0.1061

The three rows represent three separate sources of information. With the subscriber record alone, the result goes 0.0318 above the baseline. When the three numbers produced by the pretext task are added, the difference rises to 0.0504; the contribution is 0.0186 and no label was spent on it. The nine hand-derived features still give 0.1061 — self-supervision has not replaced domain knowledge, it has closed a quarter of the gap.

The label accounting is written down too. The model of the pretext task was built only with the readings of the training set; not a single row of the 377 subscribers in the held-out set entered the computation of the coefficients. In the main task all three rows use the same 883 labels, which means the difference between the rows comes not from the number of labels but only from the feature set. The cost of the representation is zero labels, and this is what separates it from the nine hand-derived features: to write those nine columns, someone had to look at the consumption series and decide which summary would be useful.

Where the Contribution Comes From

The table read up to here seems to say this: the pretext task learned something and what it learned was carried to the main task. This claim can be tested, because the baseline of the pretext task produces a representation too. The overall mean is put in place of the prediction, the same three numbers are computed again and they are measured in the same place. This representation, which carries nothing learned, shows how much of what was learned really comes from learning.

# audit.py — the baseline of the pretext task produces a representation too: the overall mean
# is put in place of the prediction. The two representations are compared in the same place.
for a in SUBSCRIBERS:
    a["expected_base"], a["deviation_base"] = OVERALL, a["series"][-1] - OVERALL
    a["relative_deviation_base"] = (a["series"][-1] - OVERALL) / (OVERALL + 1.0)
BASE_REPRESENTATION = ["expected_base", "deviation_base", "relative_deviation_base"]
print("feature set".ljust(38) + "count".rjust(7) + "train".rjust(12)
      + "held-out".rjust(12) + "vs baseline".rjust(13))
measure("record", RECORD)
measure("record + baseline representation", RECORD + BASE_REPRESENTATION)
measure("record + learned representation", RECORD + REPRESENTATION)

feature set                             count       train    held-out  vs baseline
record                                      3      0.7939      0.7851      +0.0318
record + baseline representation            6      0.8301      0.8064      +0.0531
record + learned representation             6      0.8313      0.8037      +0.0504

The claim collapsed. The representation built from the baseline gives 0.8064, the learned representation gives 0.8037; the contribution of the learned part is minus 0.0027. What really contributes to the main task is not the relationship the pretext task learned but the raw observation that the representation carries along. Because the overall mean is constant, the three columns in the second row are in fact three different ways of writing the last reading, and giving the decision tree the last reading as a new column explains the whole contribution.

The consequence of this is at the curriculum level. The 0.4115 m³ gained on the pretext task is not the number gained on the main task. The two tasks have two separate measures, two separate baselines and two separate results; beating the baseline in one does not mean beating it in the other. To say that a pretext task works, one must write not the gain on its own measure but the difference that it adds in the main task over the representation of the baseline.

It can also be written why the fictional dataset behaves this way. In this dataset the periods of the same subscriber are generated largely independently of one another; the reading of one period weakly determines the next, and the coefficient of the earlier mean stays at 0.2620. In a dataset strongly tied over time, the same pretext task would give a different number. What is measured is not the method itself but the contribution of the method on this data.

Summary

• In the self-supervised setup the label does not come from outside, it is derived from the data: the last reading is hidden and predicted from the earlier readings.
• The pretext task beats the baseline on its own measure: on the held-out set the mean absolute error falls from 6.3776 m³ to 5.9661 m³, a gain of 0.4115 m³ with zero labels spent.
• When the three numbers produced by the pretext task are carried to the main task, the 0.7851 measured with the record features alone rises to 0.8037; the nine hand-derived features give 0.8594.
• When the same representation is built from the baseline of the pretext task, it gives 0.8064: the contribution of the learned part is minus 0.0027 and the difference comes from the raw observation carried along.
• Beating the baseline in the pretext task does not mean beating the baseline in the main task; the two tasks have two separate measures and two separate baselines.

Next Step

In three lessons a single question was asked: is this subscriber suspicious. Because the question is binary, the baseline also stayed the same throughout — telling the most frequent class. Yet the same table can carry other questions too: how many cubic meters this subscriber will consume next period, into how many natural groups the subscribers split, which hundred subscribers the team going out to the field today should visit and in what order. The next lesson asks the same data four separate questions and chooses the baseline of each question separately: four separate non-learning procedures are written for classification, regression, clustering and ranking, the reasons for their choice are recorded and it is shown that the four baselines give four separate numbers.