Error Analysis

The previous lesson ended with a number: of the 252 rows of the test set, the right decision in 201 and the wrong one in 51. Those 51 rows were squeezed into a single ratio throughout the course; which regions, which tariff steps, which household sizes and how many readings they came from was not asked.

Error analysis asks this question: is the error distributed equally over the set, or is it concentrated in one slice. The question has a trap and this lesson measures that too: if enough slices are defined, some of them will look bad by chance.

• ET56. The model is in the form chosen by the previous lesson: degree 1, nine columns. The penalty and the threshold are chosen with the same seed, with a 200-candidate search and only from the validation set. The analysis is done on the test set and no choice is made from it.
• ET57. The slice axes are four: region (five values), tariff step (four intervals with the limits of 10, 25 and 40 m³), household size (1–2, 3–4, 5–6) and number of readings (1, 2, 3). A slice that does not carry at least 10 rows counts as not measurable; the threshold is fixed before the measurement.
• ET58. The baseline inside a slice is the same procedure as the baseline of the course: saying the most frequent class of the training set. Because the class composition changes, the baseline varies from slice to slice.
• ET59. The measure of chance is the reshuffling of the right/wrong decisions across the rows; the procedure was established in the Statistical Analysis and Exploratory Data Analysis course and is not repeated.
• ET60. This lesson measures the performance difference across slices. What the difference does to whom and for whom the model is built is left to the Model Evaluation, Interpretability and Ethics course.

Where the Error Concentrates

# COMMON — MODEL: The set and the model come from the FICTIONAL subscriber table of the
# Introduction to Machine Learning course; the tuning is chosen with the 200-candidate search of
# the previous lesson and from the validation set only.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(items, seed):
    r, s = rng(seed), list(range(len(items)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [items[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "zero_readings", "volatility", "household", "satisfaction",
          "period", "area_east", "area_west", "area_south"]


def solve(A, b):
    n = len(A)
    M = [list(A[i]) + [b[i]] for i in range(n)]
    for i in range(n):
        p = max(range(i, n), key=lambda r: abs(M[r][i]))
        M[i], M[p] = M[p], M[i]
        for r in range(n):
            if r != i:
                f = M[r][i] / M[i][i]
                for c in range(i, n + 1):
                    M[r][c] -= f * M[i][c]
    return [M[i][n] / M[i][i] for i in range(n)]


def fit(s, penalty):
    d = len(FIELDS) + 1
    A = [[0.0] * d for _ in range(d)]
    b = [0.0] * d
    for x in s:
        v = [1.0] + [float(x[a]) for a in FIELDS]
        for i in range(d):
            b[i] += v[i] * x["suspicious"]
            for j in range(i, d):
                A[i][j] += v[i] * v[j]
    for i in range(d):
        for j in range(i):
            A[i][j] = A[j][i]
        if i:
            A[i][i] += penalty
    return solve(A, b)


CACHE = {}


def decisions(candidate, subset):
    penalty, threshold = candidate
    if penalty not in CACHE:
        CACHE[penalty] = fit(TRAIN, penalty)
    w = CACHE[penalty]
    return [1 if w[0] + sum(w[i + 1] * float(x[a]) for i, a in enumerate(FIELDS))
            >= threshold else 0 for x in subset]


r = rng(SEED + 4100)
CANDIDATES = [(10 ** (-3 + 6 * r()), 0.20 + 0.40 * r(), r(), r()) for _ in range(200)]
CHOSEN, BEST = None, -1.0
for a in CANDIDATES:
    d = sum(k == x["suspicious"] for k, x in zip(decisions(a[:2], VAL), VAL)) / len(VAL)
    if d > BEST:
        BEST, CHOSEN = d, a[:2]

MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN))
CORRECT = [k == x["suspicious"] for k, x in zip(decisions(CHOSEN, TEST), TEST)]
BASELINE = [MAJORITY == x["suspicious"] for x in TEST]
OVERALL = sum(CORRECT) / len(TEST)
print(f"chosen candidate: penalty {CHOSEN[0]:.4g}, threshold {CHOSEN[1]:.4f}, "
      f"validation {BEST:.4f}")
print(f"test set {len(TEST)} rows, model {OVERALL:.4f}, "
      f"baseline {sum(BASELINE) / len(TEST):.4f}, errors {len(TEST) - sum(CORRECT)}")

chosen candidate: penalty 0.00372, threshold 0.5540, validation 0.8452
test set 252 rows, model 0.7976, baseline 0.7579, errors 51

Below, the 51 errors are broken down along four axes. The number of the model, the baseline in the slice, their difference and two shares are printed: the error share (how many of the errors are there) and the row share (how many of the rows are there). When the two shares part, the error is concentrated.

AXIS = {"region": lambda x: x["region_name"],
        "household": lambda x: ("1-2", "3-4", "5-6")[min(2, (x["household"] - 1) // 2)],
        "readings": lambda x: str(x["period"]),
        "tariff": lambda x: ("<10", "10-25", "25-40", ">=40")[
            0 if x["mean_consumption"] < 10 else 1 if x["mean_consumption"] < 25
            else 2 if x["mean_consumption"] < 40 else 3]}
MIN_ROWS = 10


def slice_up(sets):
    d = {}
    for axes in sets:
        for i, x in enumerate(TEST):
            d.setdefault((axes, tuple(AXIS[e](x) for e in axes)), []).append(i)
    return d


SINGLE = [("region",), ("tariff",), ("household",), ("readings",)]
ERRORS = len(TEST) - sum(CORRECT)
print(f"{'axis':<9} {'slice':<7} {'n':>4} {'model':>7} {'baseline':>8} {'difference':>10} "
      f"{'error share':>11} {'row share':>9}")
for k, idx in sorted(slice_up(SINGLE).items()):
    if len(idx) < MIN_ROWS:
        print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4}  not measurable (below {MIN_ROWS} rows)")
        continue
    m = sum(CORRECT[i] for i in idx) / len(idx)
    t = sum(BASELINE[i] for i in idx) / len(idx)
    h = sum(1 for i in idx if not CORRECT[i])
    print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4} {m:>7.4f} {t:>8.4f} {m - t:>+10.4f} "
          f"{h / ERRORS:>11.3f} {len(idx) / len(TEST):>9.3f}")

axis      slice      n   model baseline difference error share row share
household 1-2       73  0.8219   0.7534    +0.0685       0.255     0.290
household 3-4      143  0.7972   0.7692    +0.0280       0.569     0.567
household 5-6       36  0.7500   0.7222    +0.0278       0.176     0.143
readings  1         22  0.8636   0.8636    +0.0000       0.059     0.087
readings  2         63  0.7460   0.7619    -0.0159       0.314     0.250
readings  3        167  0.8084   0.7425    +0.0659       0.627     0.663
region    central   58  0.6897   0.6552    +0.0345       0.353     0.230
region    east      36  0.6944   0.4722    +0.2222       0.216     0.143
region    north     68  0.8088   0.8235    -0.0147       0.255     0.270
region    south     58  0.8448   0.8276    +0.0172       0.176     0.230
region    west      32  1.0000   1.0000    +0.0000       0.000     0.127
tariff    10-25    194  0.8402   0.8402    +0.0000       0.608     0.770
tariff    25-40     46  0.6522   0.4565    +0.1957       0.314     0.183
tariff    <10       11  0.6364   0.6364    +0.0000       0.078     0.044
tariff    >=40       1  not measurable (below 10 rows)

Of fifteen slices fourteen are measurable; in the highest tariff step there is a single subscriber and the table states that nothing can be said about that step.

There are two negatives in the difference column: in the north region the model is 0.8088 against a baseline of 0.8235; among two-reading subscribers it is 0.7460 against 0.7619. The overall number of 0.7976 says nothing in these two slices.

Concentration is read in the comparison of the two shares. The central region holds 0.230 of the rows while carrying 0.353 of the errors, that is, its error share is one and a half times its row share; the 25–40 m³ step is 0.314 against 0.183. The west region holds 0.127 of the rows and produces no errors at all — there is not a single subscriber with a suspicious label there, the baseline is 1.0000 and the best the model can do is to stay equal to the baseline. The east region and the 25–40 m³ step are both the places where the model gains most over the baseline (+0.2222 and +0.1957) and the places that are worst in absolute terms (0.6944 and 0.6522).

As the Number of Slices Grows

If the axes are crossed, the number of slices grows quickly and so does the number of “bad slices” to be found. The question is how much of this growth is real. The measure is this: the right and wrong decisions are reshuffled across the rows — the number of errors stays fixed at 51, the tie to the slice structure is broken — and the count is repeated.

PAIRS = SINGLE + [("region", "tariff"), ("region", "readings"),
                  ("tariff", "household"), ("household", "readings")]
TRIPLES = PAIRS + [("region", "tariff", "readings")]


def count_bad(d, correct):
    measurable = bad = 0
    for idx in d.values():
        if len(idx) < MIN_ROWS:
            continue
        measurable += 1
        if (sum(correct[i] for i in idx) / len(idx)
                < sum(BASELINE[i] for i in idx) / len(idx)):
            bad += 1
    return measurable, bad


print(f"{'slice set':<14} {'defined':>8} {'measurable':>10} {'bad':>5} "
      f"{'chance mean':>12} {'chance max':>11} {'baseline>model':>14}")
for name, subset in (("single axis", SINGLE), ("two axes", PAIRS), ("three axes", TRIPLES)):
    d = slice_up(subset)
    measurable, bad = count_bad(d, CORRECT)
    chance = [count_bad(d, shuffle(CORRECT, SEED + 8000 + i))[1] for i in range(200)]
    above = sum(1 for idx in d.values() if len(idx) >= MIN_ROWS
                and sum(BASELINE[i] for i in idx) / len(idx) > OVERALL)
    print(f"{name:<14} {len(d):>8} {measurable:>10} {bad:>5} "
          f"{sum(chance) / len(chance):>12.2f} {max(chance):>11} {above:>14}")

worst = min(((sum(CORRECT[i] for i in idx) / len(idx)
              - sum(BASELINE[i] for i in idx) / len(idx), k, len(idx))
             for k, idx in slice_up(PAIRS).items() if len(idx) >= MIN_ROWS))
print(f"\nworst two-axis slice: {worst[1][0]} = {worst[1][1]}, "
      f"{worst[2]} rows, difference {worst[0]:+.4f}")

slice set       defined measurable   bad  chance mean  chance max baseline>model
single axis          15         14     2         4.41           6              5
two axes             62         41     6        14.26          18             16
three axes           92         50     9        18.46          23             21

worst two-axis slice: ('region', 'tariff') = ('east', '10-25'), 17 rows, difference -0.1176

The number of bad slices goes from 2 to 6 and from 6 to 9. Read alone, this looks like proof that the model falls flat in more places as the slices are thinned. The chance columns say the opposite: when the same 51 errors are distributed randomly over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, and the observed number is below this average in all three sets.

The last column says where this comes from. The overall number of the model is 0.7976; in a slice whose baseline is above this number, a model whose errors are randomly distributed falls below the baseline. Such slices number 5 on a single axis, 16 on two axes and 21 on three axes, and the chance averages sit right next to these numbers. The “bad slice” criterion is really counting the slices whose baseline is higher than the overall performance.

The observed staying below chance is in the model’s favor. This is the form here of the rule measured in the Statistical Analysis and Exploratory Data Analysis course — a number is not a finding until it is measured how often the same number would be found by chance. What the differences found mean for whom is left to the Model Evaluation, Interpretability and Ethics course.

Summary

• Of fifteen slices fourteen are measurable; in the highest tariff step there is a single subscriber and nothing can be said about that step.
• In two slices the model is below the baseline: the north region (0.8088 against 0.8235) and two-reading subscribers (0.7460 against 0.7619).
• The central region holds 0.230 of the rows and carries 0.353 of the errors; the east region is both the slice that gains most over the baseline (+0.2222) and the one that is worst in absolute terms (0.6944).
• As the number of slices goes from 14 to 50, the bad slices go from 2 to 9, but with randomly distributed errors they go from 4.41 to 18.46; the observed number is below chance in every set.

Course Wrap-Up

The account of twenty lessons is in a single table, and every number was read from its own lesson. The second column is, in every lesson, the test-set number of the candidate selected from the validation set. The classification rows are accuracy, with a baseline of 0.7579; the regression rows are root mean squared error (m³), with a baseline of 6.5324.

Lesson	Number of the family	Candidates tried	Above baseline
Linear 01	5.3096 m³	1	+1.2228
Linear 02	5.3775 m³	8	+1.1549
Linear 03	5.2954 m³	18	+1.2370
Linear 04	0.7937	9	+0.0357
Linear 05	0.8056	8	+0.0477
Nonlinear 01	0.7937	24	+0.0357
Nonlinear 02	0.7817	20	+0.0238
Nonlinear 03	0.7857	17	+0.0278
Nonlinear 04	0.7857	12	+0.0278
Nonlinear 05	0.7897	31	+0.0317
Nonlinear 06	0.7738	1,279	+0.0159
Nonlinear 07	0.7778	3	+0.0199
Evaluation 01	0.8016	49	+0.0437
Evaluation 02	0.8016	5	+0.0437
Evaluation 03	181 cost	99	−185 cost
Evaluation 04	0.0242 calibration	5	−0.0044
Evaluation 05	0.4770 explained variation	3	+0.4770
Evaluation 06	0.8056	1	+0.0477
Evaluation 07	0.7976	200	+0.0397
Evaluation 08	0.7976	1	+0.0397

Three rows are read in their own units. The unit of Evaluation 03 is cost and smaller is better: 181 units, 185 below the baseline’s 366. Evaluation 04 is the calibration error and is the only negative row in the table: the baseline that says a constant number is better at 0.0198 than the model’s 0.0242, that is, the learned model could not beat the unlearned procedure there.

The rule of the course is the third column: the number of a model cannot be read without writing how many candidates were tried to find that number; a number whose tuning budget is not written counts as unmeasured. When the column is summed, the number of candidates tried in the course comes out: 1,793.

The second claim of the course is read from the table. The best of the classification rows is 0.8056, the worst 0.7738: all seven families are inside a band of 0.0318 points. The difference inflated by the tuning budget is the size of this band — in Evaluation 07, 0.0476 between validation and test at budget 200, and in the lesson with the table’s highest budget, 1,279 candidates, 0.0318 between two choices. The whole difference among the seven families is equal to the difference produced by the tuning budget of a single lesson.

In twenty lessons seven model families were tried and the budget was written next to every number. But all of these measurements had a common support: the label. The baseline said the most frequent label, the measure compared the predicted label with its truth, the search looked at the labels on the validation set. In a set where there is no label, what is learned — even whether anything is learned — is read against which baseline, was never asked. The next course, Unsupervised Learning, opens with this question.