Reproducibility

The previous lesson counted that a statement closes eighteen of twenty-two questions. The number the statement carried came from a single run: one seed, one split, one depth, one threshold. Nowhere was it shown that the same number would come out when that run is made a second time.

This lesson prints two numbers. The first is the total number of items: how many decisions must be written down for the same test count to be reproduced. The second is the number a single unfixed item moves. The item count is inherited from the Data Collection and Preparation, Data Analysis with Python, Visualization, Reporting and Analysis at Scale, Data Preparation and Feature Engineering and Deep Learning courses; this lesson only adds the evaluation’s own items and gives the total.

• ER38. The set, label and model are inherited from the first seven courses of the Machine Learning and Data Science curriculum and are fictional; the preparation items were fixed in the Data Collection and Preparation course and stay fixed in this lesson.
• ER39. Releasing an item does not mean deleting it but moving it to another equally defensible value. The alternatives are fictional and none of them is wrong.
• ER40. Two numbers are read in every run: the accuracy measured overall on the test set and the subgroup gap measured on the four axes of the previous lesson.
• ER41. Items are released one at a time; while one is released, the other eight stay fixed.
• ER42. When the shift is converted to a number of subscribers, that run’s own test set size is used, because the set size also changes when the split item is released.
• ER43. Floating-point comparison is made with a threshold of 1e-9; this threshold is far below an accuracy difference rounded to four decimals.

The Evaluation’s Own Items

# reproducibility.py — MODEL: The FICTIONAL subscriber table is built with the same seed (20260218);
# the preparation items were fixed in the Data Collection and Preparation course and stay fixed here.
# The evaluation's own items are collected in a single dictionary, and the run reads it.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings",
          "area_north", "area_south", "area_east", "area_west", "area_central"]
DEFAULTS = {"split_seed": 90000, "train_share": 0.60, "depth": 6,
            "leaf_min": 2, "leaf_threshold": 0.5, "candidate_count": 10,
            "subgroup_min": 20, "tiers": (10, 25, 40), "field_order": "natural"}


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


DATA = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() < 0.046:
        continue
    num = 10001 + i
    r, v = rng(SEED + 7000 + num), []
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            b[2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + num)
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3]
         + 0.9 * (num - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": household, "satisfaction": rating, "period": len(v), "region": b[0],
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    for q in REGION:
        x["area_" + q[0]] = int(b[0] == q[0])
    DATA.append(x)


def run(**released):
    """Reads the item dictionary and returns the test accuracy and the subgroup gap."""
    a = dict(DEFAULTS, **released)
    r, s = rng(SEED + a["split_seed"]), list(DATA)
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    n = int(len(s) * a["train_share"])
    train, held = s[:n], s[n + (len(s) - n) // 2:]
    fields = FIELDS if a["field_order"] == "natural" else FIELDS[::-1]

    def tree(u, depth):
        p, best = sum(x["suspicious"] for x in u) / len(u), None
        if depth and len(u) >= 2 * a["leaf_min"] and 0.0 < p < 1.0:
            g = 2 * p * (1 - p)
            for field in fields:
                d = sorted({x[field] for x in u})
                for v in (d[1:] if len(d) < a["candidate_count"] else
                          [d[int(i * len(d) / a["candidate_count"])]
                           for i in range(1, a["candidate_count"])]):
                    left = [x for x in u if x[field] < v]
                    right = [x for x in u if x[field] >= v]
                    if min(len(left), len(right)) < a["leaf_min"]:
                        continue
                    q = 0.0
                    for y in (left, right):
                        w = sum(x["suspicious"] for x in y) / len(y)
                        q += len(y) * 2 * w * (1 - w) / len(u)
                    if best is None or g - q > best[0]:
                        best = (g - q, field, v, left, right)
        if best is None or best[0] <= 1e-9:
            return {"prediction": int(p > a["leaf_threshold"])}
        return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1),
                "right": tree(best[4], depth - 1)}

    M = tree(train, a["depth"])

    def accuracy(u):
        hits = 0
        for x in u:
            k = M
            while "prediction" not in k:
                k = k["left"] if x[k["field"]] < k["threshold"] else k["right"]
            hits += k["prediction"] == x["suspicious"]
        return hits / len(u)

    t1, t2, t3 = a["tiers"]
    axes = [lambda x: x["region"],
            lambda x: "tier %d" % (1 + (x["mean_consumption"] >= t1)
                                   + (x["mean_consumption"] >= t2)
                                   + (x["mean_consumption"] >= t3)),
            lambda x: "1-2" if x["household"] <= 2 else
            ("3-4" if x["household"] <= 4 else "5+"),
            lambda x: "few" if x["period"] <= 1 else "full"]
    measured = []
    for f in axes:
        for g in sorted({f(x) for x in held}):
            sub = [x for x in held if f(x) == g]
            if len(sub) >= a["subgroup_min"]:
                measured.append(accuracy(sub))
    return (accuracy(held), max(measured) - min(measured), len(measured),
            len(held))


BASE, GAP, COUNT, N = run()
print(f"with all items fixed, test {BASE:.4f}, subgroup gap {GAP:.4f}; "
      f"{COUNT} measured subgroups in the test set of {N} subscribers")

with all items fixed, test 0.7778, subgroup gap 0.3576; 12 measured subgroups in the test set of 252 subscribers

None of the nine items is hidden inside a line of code; all of them sit in a single dictionary that the run function reads. This is the first condition of fixing: whether an item buried in the code is fixed or released cannot be asked.

A Single Unfixed Item

# sweep.py — each item is released ALONE: it is moved to another equally defensible value
# while the others stay fixed. Two numbers are read at once: the test accuracy
# measured overall and the subgroup gap.
RELEASED = [("split seed", "split_seed", 90001),
            ("train share", "train_share", 0.70),
            ("tree depth", "depth", 5),
            ("minimum subscribers in a leaf", "leaf_min", 5),
            ("leaf threshold", "leaf_threshold", 0.45),
            ("candidate threshold count", "candidate_count", 8),
            ("minimum subgroup size", "subgroup_min", 30),
            ("tariff tier boundaries", "tiers", (12, 28, 45)),
            ("feature list order", "field_order", "reversed")]

print(f"{'released item':<32}{'new value':<14}{'test':>10}"
      f"{'subscribers':>13}{'gap':>8}{'shift':>9}{'subgroups':>11}")
SHIFTS = []
for label, key, value in RELEASED:
    s, c, k, n = run(**{key: value})
    SHIFTS.append((label, abs(s - BASE), abs(c - GAP), n))
    print(f"{label:<32}{str(value):<14}{s:>10.4f}{round(abs(s - BASE) * n):>13}"
          f"{c:>8.4f}{c - GAP:>+9.4f}{k:>11}")
moved = sum(1 for _, u, _, _ in SHIFTS if u > 1e-9)
top_acc = max(SHIFTS, key=lambda t: t[1])
top_gap = max(SHIFTS, key=lambda t: t[2])
print(f"\nof {len(RELEASED)} items, {moved} move the test count on their own; "
      f"largest shift {top_acc[1]:.4f} ({top_acc[0]}), {round(top_acc[1] * top_acc[3])} subscribers")
print(f"{sum(1 for _, _, u, _ in SHIFTS if u > 1e-9)} of the same items move the subgroup "
      f"gap; largest shift {top_gap[2]:.4f} ({top_gap[0]}), "
      f"{top_gap[2] / top_acc[1]:.1f} times the largest shift read overall")

released item                   new value           test  subscribers     gap    shift  subgroups
split seed                      90001             0.8056            7  0.3373  -0.0204         12
train share                     0.7               0.8148            7  0.4286  +0.0709         11
tree depth                      5                 0.7897            3  0.4565  +0.0989         12
minimum subscribers in a leaf   5                 0.7778            0  0.3601  +0.0024         12
leaf threshold                  0.45              0.7817            1  0.3723  +0.0146         12
candidate threshold count       8                 0.7579            5  0.4470  +0.0894         12
minimum subgroup size           30                0.7778            0  0.3576  +0.0000         11
tariff tier boundaries          (12, 28, 45)      0.7778            0  0.3854  +0.0278         13
feature list order              reversed          0.7817            1  0.3601  +0.0024         12

of 9 items, 6 move the test count on their own; largest shift 0.0370 (train share), 7 subscribers
8 of the same items move the subgroup gap; largest shift 0.0989 (tree depth), 2.7 times the largest shift read overall

Six of the nine items move the test count on their own, and the largest shift is seven subscribers. Seven subscribers may look small; yet the model’s whole difference above the baseline is five subscribers. The sixth row writes this most sharply: when the candidate threshold count is 8 instead of 10, the test accuracy is 0.7579, that is, exactly the baseline. A single unjustified item turns a model that beats the baseline into a model that does not.

The two columns on the right are the course’s own measure, and a harsher one. The largest shift in the gap is 0.0989, 2.7 times what is read overall. The subgroup gap is more fragile, because it is measured on small sets and read from the extremes.

The last column is the invisible face of fixing. In the “minimum subgroup size” row neither headline number moves at all, but the measured subgroups fall from 12 to 11; when the tariff tier boundaries are released they rise to 13. An item can change what is measured without changing the reported number, and the previous lesson counted how that spoils the decision.

The Full Item List of the Machine Learning and Data Science Curriculum

# total.py — the full item list of the curriculum. The first five sources were COUNTED IN EARLIER COURSES
# and are not recomputed here; this lesson only adds the evaluation's items
# and gives the total.
SOURCES = [("preparation notebook (Data Collection and Preparation)", 39),
           ("working environment (Data Analysis with Python)", 9),
           ("the report itself (Visualization, Reporting and Analysis at Scale)", 11),
           ("preparation chain (Data Preparation and Feature Engineering)", 4),
           ("training itself (Deep Learning)", 8),
           ("evaluation itself (this lesson)", len(RELEASED))]
for label, n in SOURCES:
    print(f"  {label:<68}{n:>3} items")
print(f"  {'total':<68}{sum(n for _, n in SOURCES):>3} items")
print(f"the moment one of the eighty items is not fixed, the number read overall "
      f"can drift by {round(top_acc[1] * top_acc[3])} subscribers and the subgroup gap by {top_gap[2]:.4f}")

  preparation notebook (Data Collection and Preparation)               39 items
  working environment (Data Analysis with Python)                       9 items
  the report itself (Visualization, Reporting and Analysis at Scale)   11 items
  preparation chain (Data Preparation and Feature Engineering)          4 items
  training itself (Deep Learning)                                       8 items
  evaluation itself (this lesson)                                       9 items
  total                                                                80 items
the moment one of the eighty items is not fixed, the number read overall can drift by 7 subscribers and the subgroup gap by 0.0989

Eighty items make a single list, and they were counted piece by piece in six separate courses. Its length is not a flaw but a measure, and none of these decisions writes itself.

Summary

• The evaluation’s own items number nine and all of them sit in a single dictionary; whether an item buried in the code is fixed or released cannot be asked.
• Six of the nine items move the test count on their own; the largest shift is seven subscribers with 0.0370, and this is larger than the model’s five-subscriber difference above the baseline. When the candidate threshold count is 8, the test accuracy falls to exactly the baseline, 0.7579.
• Eight of the same items move the subgroup gap; the largest shift is 0.0989, 2.7 times what is read overall, because the gap is read on small sets and from the extremes.
• When the minimum subgroup size is released, neither headline number moves but the measured subgroups fall from 12 to 11: an item can change what is measured without changing the reported number.
• On top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course, the 11 of the Visualization, Reporting and Analysis at Scale course, the 4 of the Data Preparation and Feature Engineering course and the 8 of the Deep Learning course, the evaluation’s 9 items are added and the full list of the curriculum comes to 80.

Course Wrap-Up

In each of the fifteen lessons the same number was read twice: once overall, once in subgroups. Because the rows do not share a unit, the cells state by which criterion they were read, and every number is taken from its own lesson.

Lesson	Overall	In the worst subgroup	Gap
Evaluation 01 Validation Strategies	accuracy 0.7778, baseline +0.0198	0.6111	0.3576
Evaluation 02 Metric Selection	cost gain +0.0198 or +0.1984	−0.4468	0.8606
Evaluation 03 Baseline Model	against eight baselines −0.0119 … +0.5357	−0.0833 in the east	0.5476 moved by the choice
Evaluation 04 Statistical Significance	difference +0.0198, its margin includes zero	−0.0471 in few readings	4 of 13 differences exclude zero
Evaluation 05 Distribution Shift	difference above the baseline 0.0198 → 0.0066	none below the baseline	0.3576 → 0.2573
Interpretability 01 The Need for Interpretability	closed questions 8.58/12	7.25 questions	1.64 questions
Interpretability 02 Feature Importance	the ranking diverges in 10 of 12 columns	0.0162 in the most important column	0.1720
Interpretability 03 Local Explanation Methods	faithfulness 0.9417	0.9027	0.0637
Interpretability 04 Partial Dependence and Interaction	household slope −0.0069 (flat)	−0.1891 in the east	0.3192
Interpretability 05 Inherently Interpretable Models	rule list 0.7897, 0.0159 to the family	0.0862 away in the central region	6 subgroups cannot get within 0.02
Ethics 01 Sources of Bias	accuracy 0.8194, recall 0.3704	0.7108 in the east	0.2733
Ethics 02 Fairness Metrics	the best attempt leaves a gap of 0.2727	base rate 0.4578 in the east	recall 0.4842, 0.7038 under predictive parity
Ethics 03 Privacy-Preserving Learning	trace of a single record 0.0313 / 0.7778	records standing alone with the sixth field 0.9857	cost of noise 0.0022 against 0.1669
Ethics 04 Model Documentation	accuracy 0.7778; the statement closes 18/22 questions	0.6111 (east)	0.3576
Ethics 05 Reproducibility	0.7778; a single item moves 7 subscribers	0.7579 at candidate threshold count 8	0.3576; a single item moves it by 0.0989

The third column is the rule of the course: a number cannot be read without writing for whom it was measured; a performance whose subgroup is not written is counted as unmeasured. In no row did the gap come out zero.

The eight courses of the Machine Learning and Data Science curriculum read the same axis in eight separate places.

Course	Lessons	Measure axis	Unit
Introduction to Machine Learning	11	separation	in training and in the held-out set
Data Preparation and Feature Engineering	12	the difference a feature adds	contribution
Supervised Learning	20	what the family adds and what the budget inflates	candidate tried
Unsupervised Learning	11	what the baseline is when there are no labels	baseline built
Deep Learning	24	what capacity adds and the share of training	second run
Natural Language Processing and Computer Vision	14	the domain’s own baseline	the domain’s baseline
Reinforcement Learning and Recommender Systems	14	the decision changing its own data	data collected
Model Evaluation, Interpretability and Ethics	15	baseline for whom	subgroup gap
Total	121

One hundred twenty-one lessons established a single rule eight times: a model is as good as how far it beats a dumber procedure; a performance whose baseline is not written is counted as unmeasured. The rule was paid in the same form in every course: next to the number of the model went the number of a dumber procedure, and where the difference came from was questioned.

The second claim was paid in every course too: the difference does not always come from learning. Leakage, selection share, tuning budget, imbalanced classes and, in the last course, the definition of the subgroup itself produced a difference.

In one hundred twenty-one lessons a model was built, measured, explained, and what it does to whom was counted. But all of it happened in a single run, on a single machine, inside a notebook. The model becoming a system — taking a version, gaining an interface that others call, being rolled back when it breaks and having its cost counted — was never taken up. The next curriculum, AI Engineering, opens with this question.