Lesson 11 / 15

Sources of Bias

Counting separately that the same harm arises in three different places: the linear model measured on thirteen subgroups gives 0.8194 accuracy and 0.3704 recall overall, while recall stays at 0.2222 for subscribers with few readings. Because of the 69 subscribers who never enter the table, the share of few-reading subscribers reads 0.3349 instead of 0.3695; a training set built only from full-reading subscribers lowers precision in that subgroup from 0.5455 to 0.3774, the 19 suspicious cases that never reach the record drop recall to 0.0370, and a capacity of forty visits applied to the same subgroup at a rate of 0.0359 leaves recall at 0.1111. In all three, overall accuracy moves by at most 0.0337.

Published October 7, 2026 Updated October 21, 2026

In this lesson

Defines data, label and deployment bias separately and counts each one on the same fictional set
Shows that the three biases hit the number of a single subgroup by different paths: for subscribers with few readings, recall goes in three different directions through three different mechanisms
Compares how much of the movement in a subgroup is hidden by the number measured overall

Contents

The previous topic opened up the number the model produces over ten lessons: what the number is, who it is for and what the model based it on were measured. The explanation made a decision readable, but a readable decision is not a good decision. An explanation says which feature a decision rests on; it does not say what the decision does to whom, and a model that systematically harms a subgroup can be explained with the same clarity.

This topic fills that gap, and the first question is the most concrete one: where the harm comes from. Who entered the table is one bias, whom the label missed is a second, and whom the model was applied to is a third. The three arise in different places, call for different fixes and, as this lesson will show, hit the same number. The bias that arises from how an observation is selected was measured in the Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course; it is not repeated here, it is taken as input.

ER1. The set, the label, the seed and the split are inherited from the Introduction to Machine Learning through Reinforcement Learning and Recommender Systems courses and are fictional: 1,260 subscribers, a training set of 756 rows. The suspicion label is not a real inspection record.
ER2. Throughout this topic the validation and test sets are merged and read as a single held-out set of 504 subscribers; subgroup counts are already too thin at 252 subscribers. The baseline of the test set is 0.7579, and the baseline of the merged set is 0.7857.
ER3. The model is the twelve-column linear model of the Supervised Learning course; its settings are fixed and it gives 0.8016 on the test set; this is the line of comparison.
ER4. Subgroups are defined in this topic and stay fixed: region (five), tariff tier (three), household size (three), reading count (two). Thirteen subgroups are examined and this number is written in every lesson; in the held-out set, a subgroup smaller than fifty records does not enter the gap calculation.
ER5. Tariff tiers are built with thresholds of 10 and 25 m³; since two subscribers remain above 40 m³, the top tier is merged. The reading count axis is split in two: few (one or two periods) and full (three periods).
ER6. The measures are accuracy, recall and precision; their definitions were built in the Supervised Learning course and are not repeated here. In a subgroup where no flag is raised at all, precision is undefined.
ER7. On top of the inherited fiction, two layers valid only in this lesson are added for two measurements: a recording process in which part of the suspicious cases among subscribers with few readings never reach the record, and a capacity in which the field crew visits only the highest-scored.
ER8. Here bias is not an intention but a measure; all three mechanisms are by-products of ordinary decisions.

Three Biases Arise in Three Different Places

The setup generates the set, trains the linear model and measures the same model separately on thirteen subgroups.

# ethics.py — MODEL: The same FICTIONAL subscriber table, the same seed and the same split;
# the target is the illicit-use or fault suspicion label, and that label is FICTIONAL too.
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA, NO_READINGS = [], [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        NO_READINGS.append(k["region"][0])
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0),
         "suspicious": int(z > 1.35)}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, HELD_OUT = K[:756], K[756:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS}
STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS}


def scaled(x):
    return [1.0] + [(x[a] - MEAN[a]) / STD[a] for a in FIELDS]


def squash(z):
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def train_model(rows, label):            # MODEL: linear model, gradient descent
    X = [scaled(x) for x in rows]
    Y = [x[label] for x in rows]
    w = [0.0] * len(X[0])
    for _ in range(200):
        g = [0.0] * len(w)
        for i in range(len(X)):
            h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i]
            for j in range(len(w)):
                g[j] += h * X[i][j]
        for j in range(len(w)):
            w[j] -= 0.3 * g[j] / len(X)
    return w


def probability(w, x):
    return squash(sum(a * b for a, b in zip(w, scaled(x))))


def groups(x):
    t = 10 if x["mean_consumption"] <= 10 else 25 if x["mean_consumption"] <= 25 else 40
    return ["region " + x["region_name"], "tariff %d" % t,
            "household " + ("1-2" if x["household"] <= 2 else "3-4" if x["household"] <= 4 else "5+"),
            "readings " + ("few" if x["period"] <= 2 else "full")]


SUBGROUPS = sorted({g for x in HELD_OUT for g in groups(x)})
W = train_model(TRAIN, "suspicious")


def measure(w, s):                       # accuracy, recall, precision
    susp = sum(x["suspicious"] for x in s)
    flagged = [x for x in s if probability(w, x) > 0.5]
    caught = sum(x["suspicious"] for x in flagged)
    return (sum((probability(w, x) > 0.5) == x["suspicious"] for x in s) / len(s),
            caught / susp if susp else float("nan"),
            caught / len(flagged) if flagged else float("nan"))


print(f"subscribers {len(SUBSCRIBERS)}, none producing readings {len(NO_READINGS)}, "
      f"labeled {len(DATA)}, suspicious {sum(x['suspicious'] for x in DATA)}")
print(f"train {len(TRAIN)}, held-out {len(HELD_OUT)}, subgroups examined {len(SUBGROUPS)}")
print(f"baseline {sum(x['suspicious'] == 0 for x in HELD_OUT) / len(HELD_OUT):.4f}, overall"
      + "".join(f"{v:>11.4f}" for v in measure(W, HELD_OUT)) + "\n")
print(f"{'subgroup':<14}{'train':>7}{'held-out':>9}{'suspicious':>11}"
      f"{'accuracy':>11}{'recall':>11}{'precision':>11}")
for g in SUBGROUPS:
    s = [x for x in HELD_OUT if g in groups(x)]
    print(f"{g:<14}{sum(1 for x in TRAIN if g in groups(x)):>7}{len(s):>9}"
          f"{sum(x['suspicious'] for x in s):>11}"
          + "".join(f"{v:>11.4f}" for v in measure(W, s)))

subscribers 1329, none producing readings 69, labeled 1260, suspicious 276
train 756, held-out 504, subgroups examined 13
baseline 0.7857, overall     0.8194     0.3704     0.6349

subgroup        train held-out suspicious   accuracy     recall  precision
household 1-2     238      144         31     0.8264     0.3871     0.6667
household 3-4     398      289         61     0.8201     0.3607     0.6286
household 5+      120       71         16     0.8028     0.3750     0.6000
readings few      255      167         27     0.8443     0.2222     0.5455
readings full     501      337         81     0.8071     0.4198     0.6538
region central    134       97         29     0.7423     0.2759     0.6667
region east       136       83         38     0.7108     0.6842     0.6842
region north      219      144         23     0.8264     0.1739     0.4000
region south      141      117         17     0.8632     0.1176     0.6667
region west       126       63          1     0.9841     0.0000        nan
tariff 10          29       22          6     0.7273     0.0000        nan
tariff 25         590      376         54     0.8511     0.2037     0.4583
tariff 40         137      106         48     0.7264     0.6042     0.7436

The three numbers read overall look good: accuracy of 0.8194 is 0.0337 above the baseline of 0.7857, recall is 0.3704 and precision is 0.6349. On the subgroups the same model shows thirteen different behaviors: accuracy is 0.9841 in the west and 0.7108 in the east; recall is 0.6842 in the east and 0.1176 in the south. The two sharpest rows are the unmeasurable ones: in the west and in the lowest tariff tier no flag is raised at all, so precision is undefined, and that tier has only 29 rows in the training set.

This lesson follows the reading count axis. Among few-reading subscribers accuracy is 0.8443, above the overall figure, but recall is 0.2222, below the 0.4198 of the full-reading subscribers; the gap is 0.1976. High accuracy is misleading: because the share of suspicious cases is low, flagging nobody at all already gives high accuracy. The number to watch is 0.2222.

Data: Who Entered the Table

Data bias is the training set not resembling the population the model will be applied to. Its source here is the measurement network: the fewer readings a subscriber produced, the more weakly it enters the table, and one that produced none does not enter at all.

FEW = [x for x in HELD_OUT if x["period"] < 3]
FULL = [x for x in HELD_OUT if x["period"] == 3]
few_in_table = sum(1 for x in DATA if x["period"] < 3)
in_network = (few_in_table + len(NO_READINGS)) / (len(DATA) + len(NO_READINGS))
print(f"few-reading subscriber share in the table {few_in_table / len(DATA):.4f}, true share "
      f"in the network {in_network:.4f}, gap {in_network - few_in_table / len(DATA):.4f}")
ONLY_FULL = [x for x in TRAIN if x["period"] == 3]
W_FULL = train_model(ONLY_FULL, "suspicious")


def show(name, w):
    a, b = measure(w, FEW), measure(w, HELD_OUT)
    print(f"{name:<26}{a[0]:>13.4f}{a[1]:>11.4f}{a[2]:>14.4f}{b[0]:>17.4f}{b[1]:>15.4f}")


print(f"\n{'training set':<26}{'few accuracy':>13}{'few recall':>11}"
      f"{'few precision':>14}{'overall accuracy':>17}{'overall recall':>15}")
show(f"{len(TRAIN)} rows, all", W)
show(f"{len(ONLY_FULL)} rows, full-reading", W_FULL)

few-reading subscriber share in the table 0.3349, true share in the network 0.3695, gap 0.0345

training set               few accuracy few recall few precision overall accuracy overall recall
756 rows, all                    0.8443     0.2222        0.5455           0.8194         0.3704
501 rows, full-reading           0.7605     0.7407        0.3774           0.7857         0.4815

The first row gives the size of the shortfall: the share of few-reading subscribers is 0.3349 in the table and 0.3695 in the network, and the entire gap of 0.0345 comes from the 69 subscribers who never enter the table.

The second table is one step further along the same mechanism: when training is built only from the 501 full-reading rows, recall in the few-reading subgroup rises to 0.7407, but precision falls from 0.5455 to 0.3774 and accuracy from 0.8443 to 0.7605. Because the model cannot tell that subgroup apart, it flags almost all of it. The direction of the harm is up in recall and down in precision; someone looking at a single number reads this as an improvement. Overall accuracy moves by only 0.0337.

Label: Whom the Record Missed

Label bias is concerned not with who is in the table but with what is written next to them. The suspicion label is the output of a recording process, and that process does not have the same information in every subgroup.

r = rng(SEED + 62000)
missed = 0
for x in TRAIN:                          # FICTIONAL recording process: only in this measurement
    x["recorded"] = x["suspicious"]
    if x["period"] < 3 and x["suspicious"] and r() < 0.4:
        x["recorded"], missed = 0, missed + 1
positives = sum(1 for x in TRAIN if x["period"] < 3 and x["suspicious"])
print(f"few-reading training rows {sum(1 for x in TRAIN if x['period'] < 3)}, suspicious "
      f"{positives}, never reached the record {missed}; among full-reading ones, never recorded 0")
W_REC = train_model(TRAIN, "recorded")
print(f"\n{'trained label':<26}{'few accuracy':>13}{'few recall':>11}"
      f"{'few precision':>14}{'overall accuracy':>17}{'overall recall':>15}")
show("ground truth", W)
show("recorded label", W_REC)

few-reading training rows 255, suspicious 44, never reached the record 19; among full-reading ones, never recorded 0

trained label              few accuracy few recall few precision overall accuracy overall recall
ground truth                     0.8443     0.2222        0.5455           0.8194         0.3704
recorded label                   0.8323     0.0370        0.3333           0.8056         0.2778

Of the 255 few-reading rows in the training set, 44 are suspicious and 19 of them never reach the record; the missed records are 2.5% of the 756 rows in total. The effect is not small: recall in the few-reading subgroup drops to 0.0370, which means the model catches only one of the 27 suspicious cases there. Precision also falls to 0.3333; this mechanism damages both measures at once.

The real difficulty is in the measure itself. If the model were trained with the recorded label and measured with the recorded label, nothing would be visible; the 19 missed suspicious cases would have been missed in the held-out set too, and the number would have come out high. A missing row can be counted, a wrong label cannot.

Deployment: Whom the Model Was Applied To

The third bias is not in the model but in its use. Every subscriber gets a score, but the field crew cannot visit them all; it goes only to the highest-scored. The model is then applied not to everyone but to a subset.

RANKED = sorted(HELD_OUT, key=lambda x: -probability(W, x))
susp_few, susp_full = sum(x["suspicious"] for x in FEW), sum(x["suspicious"] for x in FULL)


def capacity(cap):
    top = RANKED[:cap]
    top_few = sum(1 for x in top if x["period"] < 3)
    return (top_few / len(FEW), (cap - top_few) / len(FULL),
            sum(1 for x in top if x["period"] < 3 and x["suspicious"]) / susp_few,
            sum(1 for x in top if x["period"] == 3 and x["suspicious"]) / susp_full,
            sum(x["suspicious"] for x in top) / (susp_few + susp_full))


print(f"{'capacity':>9}{'few applied':>17}{'full applied':>17}"
      f"{'few recall':>17}{'full recall':>17}{'overall recall':>17}")
for cap in (40, 60, 80):
    print(f"{cap:>9}" + "".join(f"{v:>17.4f}" for v in capacity(cap)))
c40 = capacity(40)
print(f"\n{'bias':<26}{'few recall':>15}{'overall recall':>18}"
      f"{'overall accuracy':>18}")
for name, w in (("none (base measurement)", W), ("data", W_FULL), ("label", W_REC)):
    a, b = measure(w, FEW), measure(w, HELD_OUT)
    print(f"{name:<26}{a[1]:>15.4f}{b[1]:>18.4f}{b[0]:>18.4f}")
print(f"{'deployment (capacity 40)':<26}{c40[2]:>15.4f}{c40[4]:>18.4f}{'-':>18}")

 capacity      few applied     full applied       few recall      full recall   overall recall
       40           0.0359           0.1009           0.1111           0.3210           0.2685
       60           0.0599           0.1484           0.2222           0.3951           0.3519
       80           0.0958           0.1899           0.3333           0.4568           0.4259

bias                           few recall    overall recall  overall accuracy
none (base measurement)            0.2222            0.3704            0.8194
data                               0.7407            0.4815            0.7857
label                              0.0370            0.2778            0.8056
deployment (capacity 40)           0.1111            0.2685                 -

At a capacity of forty visits the model is applied to 0.0359 of the few-reading subscribers and 0.1009 of the full-reading ones; the application rate differs by nearly a factor of three between them. The result is read from recall: 0.1111 among few-reading subscribers and 0.3210 among full-reading ones. The model, the threshold and the training set did not change; only how many people were visited changed, and the gap rose to 0.2099. At a capacity of 80 the gap falls to 0.1235: deployment bias is a by-product of capacity.

The last table is the total of the lesson. The same number of the same subgroup goes to 0.7407, 0.0370 and 0.1111 through three mechanisms — one up, two down — and overall accuracy moves by at most 0.0337. Counting the three separately is necessary, because each calls for a different fix: collecting the missing rows, auditing the recording process, distributing the capacity. Since thirteen subgroups are examined, part of these differences can also arise from the number of looks alone; the multiple comparisons lesson of the Statistical Analysis and Exploratory Data Analysis course measured its share, and this is why the subgroup count is written in every lesson.

Summary

The linear model that gives 0.8194 accuracy and 0.3704 recall overall shows thirteen different behaviors on thirteen subgroups: accuracy is 0.9841 in the west and 0.7108 in the east, and in two subgroups precision stays undefined because no flag is raised at all.
Data bias is who entered the table: because of the 69 subscribers who produced no readings, the share of few-reading subscribers reads 0.3349 instead of 0.3695, and when training is built only from full-reading subscribers, precision falls from 0.5455 to 0.3774.
Label bias is what is written next to them: the 19 suspicious cases that never reach the record drop recall to 0.0370 and are not visible without a measure outside the label.
Deployment bias is whom the model was applied to: at a capacity of forty visits the model is applied to 0.0359 of the few-reading subscribers and 0.1009 of the full-reading ones, and the gap rises to 0.2099.
While the three mechanisms carry the same number in three different directions, overall accuracy moves by at most 0.0337.

Next Step

In this lesson, when the number of one subgroup came out lower than another’s, it was assumed to be a harm and the gap was reported directly. Yet which measure a gap is measured in is a decision in its own right. Equalizing selection rates, equalizing the shares of suspicious cases caught and equalizing how many of the flagged are really suspicious are three different things. The next lesson builds these three criteria under their structural names, moves the threshold so as to fix each in turn and counts how much the others are broken. The result is not a tuning error.

To keep your progress and take notes, Log in