A/B Test Analysis

The previous lesson ran a single test: two groups, one p-value, the effect size next to it. The answer given was correct but it belonged to a single question. Once the dataset of a finished-design experiment is in hand, the question is not single. Which metric will be reported, will the breakdown be looked at, where will the observation window close, what will be done if the assignment stayed unbalanced. Every choice is defensible separately and every choice produces a different number.

The number of this lesson is the number of decisions that come out of the same experiment dataset. The design of the experiment is not set up here: how many subscribers were needed, why the experiment was not stopped early and how many comparisons tightened the limit and how were computed in the A/B and Multivariate Testing lesson of the User Experience and Behavior Design course. This lesson starts after the design is finished: the assignment is made, the readings are collected, the dataset is in hand.

• AS26. The dataset is inherited from K03 and is fictional: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written in the code.
• AS27. The experiment is fictional too: period 1 is the baseline reading, the reminder notification was sent after period 1, and the assignment is unbalanced 70/30. The assignment is drawn independently of the readings and changes no reading; the true difference between the groups is exactly zero.
• AS28. The test is produced by permutation (500 permutations), the interval by bootstrap (1,000 draws). No distribution table, critical value or ready-made test function is used.
• AS29. The “significant” threshold is 0.05. This is not a measurement but a decision, and its share is written at the end of the lesson.
• AS30. The four analytic degrees of freedom and their option counts: metric 2, breakdown 9, observation window 3, handling of the unbalanced assignment 2.

The Single Decision Written Before the Experiment

A subscriber who enters the experiment dataset is one who has a period 1 reading. The primary decision is written in a single line: metric mean consumption, no breakdown, window periods 2 and 3, assignment left raw.

# ab_test.py — MODEL. The fictional dataset of K03 is produced with the same seed, and a fictional
# experiment assignment is added on top of it. The assignment is drawn independently of the readings
# and changes no reading: the true difference between the groups is exactly zero.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DROPPED = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (DROPPED if r() < 0.046 else SUBSCRIBERS).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {k["subscriber_id"] for k in SUBSCRIBERS}
READINGS = {}
for k in SUBSCRIBERS + DROPPED:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100
            + 0.5) / 100
        if k["subscriber_id"] in VALID:
            READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t
        r()

# The experiment is FICTIONAL: period 1 is the baseline reading, the reminder notification was
# sent after period 1 and the assignment is unbalanced 70/30 (two of every three subscribers are B).
EXPERIMENT = []
for k in SUBSCRIBERS:
    d = READINGS.get(k["subscriber_id"], {})
    if 1 not in d:
        continue
    r = rng(SEED + 31000 + k["subscriber_id"])
    EXPERIMENT.append({"region": k["region"], "group": "B" if r() < 0.70 else "A",
                       "baseline_reading": d[1], "d2": d.get(2), "d3": d.get(3)})

PERMUTATIONS, RESAMPLES = 500, 1000
rs = rng(SEED + 55000)


def shuffle(items):
    h = items[:]
    for i in range(len(h) - 1, 0, -1):
        j = int(rs() * (i + 1))
        h[i], h[j] = h[j], h[i]
    return h


def permutation_test(a, b):
    """Permutation test: the labels are shuffled, the difference is recomputed."""
    na, nb = len(a), len(b)
    observed = sum(b) / nb - sum(a) / na
    exceeded = 0
    for _ in range(PERMUTATIONS):
        h = shuffle(a + b)
        if abs(sum(h[na:]) / nb - sum(h[:na]) / na) >= abs(observed) - 1e-12:
            exceeded += 1
    return observed, exceeded / PERMUTATIONS, na, nb


def interval(a, b):
    """Bootstrap: the difference is computed on a dataset redrawn 1,000 times."""
    na, nb, diffs = len(a), len(b), []
    for _ in range(RESAMPLES):
        ya = sum(a[int(rs() * na)] for _ in range(na)) / na
        yb = sum(b[int(rs() * nb)] for _ in range(nb)) / nb
        diffs.append(yb - ya)
    diffs.sort()
    return diffs[24], diffs[974]


def value(x, fields, metric):
    d = [x[a] for a in fields if x[a] is not None]
    if not d:
        return None
    return sum(d) / len(d) if metric == "mean" else (1.0 if max(d) >= 25 else 0.0)


def split_groups(sub, fields, metric):
    va, vb = [], []
    for x in sub:
        v = value(x, fields, metric)
        if v is not None:
            (va if x["group"] == "A" else vb).append(v)
    return va, vb


print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, "
      f"{sum(len(v) for v in READINGS.values())} readings")
print(f"experiment dataset: {len(EXPERIMENT)} subscribers, "
      f"A {sum(1 for x in EXPERIMENT if x['group'] == 'A')} / "
      f"B {sum(1 for x in EXPERIMENT if x['group'] == 'B')}")

# The single decision written before the experiment: metric mean, no breakdown, window
# periods 2-3, assignment left raw.
va, vb = split_groups(EXPERIMENT, ("d2", "d3"), "mean")
observed, p, na, nb = permutation_test(va, vb)
low, high = interval(va, vb)
print(f"\nprimary decision  mean consumption difference (B - A)")
print(f"  observed difference : {observed:.3f} m3   ({na} + {nb} subscribers)")
print(f"  margin of variation : [{low:.3f}, {high:.3f}] m3  (1,000 bootstrap draws)")
print(f"  p-value             : {p:.3f}  ({PERMUTATIONS} permutations)")

fictional dataset (seed 20260218): 1329 subscribers, 3199 readings
experiment dataset: 1260 subscribers, A 402 / B 858

primary decision  mean consumption difference (B - A)
  observed difference : 0.123 m3   (344 + 757 subscribers)
  margin of variation : [-0.698, 0.952] m3  (1,000 bootstrap draws)
  p-value             : 0.786  (500 permutations)

This is the line to be reported: a difference of 0.123 m³, next to it a margin of variation 1.650 m³ wide, and a p-value of 0.786. The interval contains zero; the permutation test produces a difference of the same size in four of every five runs. The number comes from the mean of 344 and 757 subscribers, and the group sizes are not equal.

The only information this line conveys is that the difference stays inside the margin of variation. This is where the experiment ends. What comes after is analysis.

Four Analytic Degrees of Freedom

On the same dataset there are four choices, and each of the four has more than one defensible value. Metric: mean consumption, or the proportion of subscribers entering the upper tariff tier. Breakdown: the whole dataset, one of the five regions, or one of three tariff tiers by baseline reading. Observation window: both periods together, period 2 only, period 3 only. Balance: the unbalanced assignment is left raw or the B group is trimmed to the size of A.

# Four degrees of freedom: the metric, the breakdown, the observation window, how the
# unbalanced assignment is handled. All the combinations are walked and it is counted how many
# come out "significant".
METRICS = ("mean", "upper tier rate")
BREAKDOWN = ("whole dataset", "north", "south", "east", "west", "central",
             "under 10", "10-25", "over 25")
WINDOWS = {"periods 2-3": ("d2", "d3"), "period 2 only": ("d2",),
           "period 3 only": ("d3",)}
BALANCE = ("raw", "trimmed")


def tier(v):
    return "under 10" if v < 10 else ("10-25" if v < 25 else "over 25")


def selected(brk):
    if brk == "whole dataset":
        return EXPERIMENT
    return [x for x in EXPERIMENT
            if x["region"] == brk or tier(x["baseline_reading"]) == brk]


SWEEP = []
for metric in METRICS:
    for brk in BREAKDOWN:
        for name, fields in WINDOWS.items():
            for balance in BALANCE:
                sub = [x for x in selected(brk) if value(x, fields, metric) is not None]
                if balance == "trimmed":
                    a = [x for x in sub if x["group"] == "A"]
                    sub = a + [x for x in sub if x["group"] == "B"][:len(a)]
                va, vb = split_groups(sub, fields, metric)
                if len(va) < 2 or len(vb) < 2:
                    continue
                SWEEP.append((metric, brk, name, balance) + permutation_test(va, vb))

significant = [s for s in SWEEP if s[5] <= 0.05]
positive = sum(1 for s in SWEEP if s[4] > 0)
means = [s[4] for s in SWEEP if s[0] == "mean"]
print(f"combinations run : {len(SWEEP)}  "
      f"({len(METRICS)} metric x {len(BREAKDOWN)} breakdown x {len(WINDOWS)} window "
      f"x {len(BALANCE)} balance)")
print(f"p <= 0.05        : {len(significant)}")
print(f"p <= 0.10        : {sum(1 for s in SWEEP if s[5] <= 0.10)}")
print(f"direction        : {positive} combinations up, {len(SWEEP) - positive} combinations down")
print(f"difference range for the mean metric: {min(means):.3f} to {max(means):.3f} m3")

print("\nmetric            breakdown   window           balance difference       p   n(A)  n(B)")
for s in sorted(significant, key=lambda s: s[5]):
    print(f"{s[0]:<18} {s[1]:>8}   {s[2]:<15}  {s[3]:<10} {s[4]:>7.3f}  {s[5]:>6.3f}  "
          f"{s[6]:>5} {s[7]:>5}")

combinations run : 108  (2 metric x 9 breakdown x 3 window x 2 balance)
p <= 0.05        : 4
p <= 0.10        : 9
direction        : 45 combinations up, 63 combinations down
difference range for the mean metric: -2.113 to 1.119 m3

metric            breakdown   window           balance difference       p   n(A)  n(B)
upper tier rate     over 25   period 3 only    trimmed     -0.195   0.006     77    77
upper tier rate     over 25   period 3 only    raw         -0.160   0.014     77   157
mean                   west   period 2 only    trimmed     -1.857   0.040     57    57
upper tier rate     central   period 3 only    trimmed     -0.189   0.040     53    53

While the true difference is zero, 4 of the 108 combinations fall below the 0.05 threshold and 9 below 0.10. All four are products of the analysis; there is no effect in the dataset that corresponds to them.

The real information of the table is not the threshold count but the distribution of two columns. In the mean metric the observed difference wanders between -2.113 and 1.119 m³; the 0.123 m³ given by the primary decision stands in the middle of this range. Of the combinations, 45 point toward an increase and 63 toward a decrease. From the same dataset both the sentences “the notification reduced consumption” and “the notification increased consumption” can be supported with numbers. The data does not speak; which combination is written speaks.

What the four significant rows have in common shows in the third and fourth columns. All four come from the narrowest breakdowns: the observation counts are between 53 and 157, whereas the primary decision worked on 1,101 subscribers. Three are in the narrowest observation window, three in the trimmed balance. In a small dataset the margin of variation grows, drawing a large observed difference from inside a large margin becomes easy, and the threshold is crossed more often. Trimming works in the same direction: reducing the B group of 858 subscribers to 402 makes the sample smaller.

Pre-registration

Since the cost of the freedom can be counted, closing it can be counted too. The four axes are written before the experiment one by one, and at every step it is looked at how many combinations remain.

# From how many to how many does pre-registration bring the
# number of combinations.
WRITTEN = (("metric", 0, "mean"), ("window", 2, "periods 2-3"),
           ("breakdown", 1, "whole dataset"), ("balance", 3, "raw"))
remaining = SWEEP
print("written before experiment   combinations left   p <= 0.05   smallest p")
print(f"{'none':<27} {len(remaining):>17}   {sum(1 for s in remaining if s[5] <= 0.05):>9}"
      f"   {min(s[5] for s in remaining):>10.3f}")
written = []
for name, i, choice in WRITTEN:
    written.append(name)
    remaining = [s for s in remaining if s[i] == choice]
    print(f"{'+ ' + name:<27} {len(remaining):>17}   {sum(1 for s in remaining if s[5] <= 0.05):>9}"
          f"   {min(s[5] for s in remaining):>10.3f}")
print(f"\nclosed combinations: {len(SWEEP)} -> {len(remaining)}  "
      f"({len(SWEEP) - len(remaining)} choices are eliminated before the experiment)")

smallest = min(SWEEP, key=lambda s: s[5])
primary = [s for s in SWEEP if s[:4] == ("mean", "whole dataset", "periods 2-3", "raw")][0]
print(f"\ntwo legitimate analyses of the same dataset:")
print(f"  {primary[0]} / {primary[1]} / {primary[2]} / {primary[3]}: "
      f"difference {primary[4]:.3f}, p {primary[5]:.3f}, {primary[6]} + {primary[7]} subscribers")
print(f"  {smallest[0]} / {smallest[1]} / {smallest[2]} / {smallest[3]}: "
      f"difference {smallest[4]:.3f}, p {smallest[5]:.3f}, {smallest[6]} + {smallest[7]} subscribers")
print("true difference: 0.000 (the assignment did not change the readings)")

written before experiment   combinations left   p <= 0.05   smallest p
none                                      108           4        0.006
+ metric                                   54           1        0.040
+ window                                   18           0        0.214
+ breakdown                                 2           0        0.758
+ balance                                   1           0        0.772

closed combinations: 108 -> 1  (107 choices are eliminated before the experiment)

two legitimate analyses of the same dataset:
  mean / whole dataset / periods 2-3 / raw: difference 0.123, p 0.772, 344 + 757 subscribers
  upper tier rate / over 25 / period 3 only / trimmed: difference -0.195, p 0.006, 77 + 77 subscribers
true difference: 0.000 (the assignment did not change the readings)

The ladder descends in four steps. Writing the metric alone brings 108 combinations to 54 and the number of combinations crossing the threshold from 4 to 1. When the observation window is written too, 18 combinations remain and none crosses the threshold; the smallest p rises from 0.006 to 0.214. When all four are written, the number of combinations is 1 and 107 choices have been eliminated before the experiment.

What is closed is not a flaw of method. Every value of the four axes is defensible on its own; the problem is that the value can be chosen after the result is seen. A decision written before the experiment reverses this order and makes the choice independent of the result.

The last block also shows the margin of the margin itself. The p-value of the primary combination came out as 0.786 in the first block and 0.772 in this one. The two numbers are two runs of the same test; the difference between them is the own margin of variation of 500 permutations. If the permutation count is increased this difference narrows, it does not vanish. If a p-value is not written together with how many permutations produced it, the third digit of that p-value should not be read.

The rule of the course grows by one item here: a p-value that does not carry next to it how many combinations were walked is not the p-value of a single test.

Summary

• The dataset of a finished-design experiment has four analytic degrees of freedom: metric (2), breakdown (9), observation window (3), handling of the unbalanced assignment (2). The number of combinations is 108.
• The single pre-registered decision gives a 0.123 m³ difference, a margin of variation of [-0.698, 0.952] m³ and a p-value of 0.786; the number comes from the mean of 344 and 757 subscribers.
• While the true difference between the groups is zero, 4 of the 108 combinations fall below 0.05 and 9 below 0.10; 45 combinations point toward an increase and 63 toward a decrease, and the difference in the mean metric wanders between -2.113 and 1.119 m³.
• The four combinations that cross the threshold rest on 53 to 157 observations; in a small breakdown the margin of variation grows and the threshold is crossed more often.
• Pre-registration brings the combinations from 108 to 1; 107 choices are eliminated and the smallest p rises from 0.006 to 0.772.
• Two runs of the same test give 0.786 and 0.772; this difference is the own margin of 500 permutations.

Next Step

This lesson counted in how many combinations a difference was measured, but what it counted was always the difference between two groups. How consumption changes not with a notification but together with a continuous variable was never asked. The next lesson models the relationship directly: least squares is written inside the lesson, a coefficient is computed, and the margin is printed next to the coefficient with bootstrap. Then a confounder is added to the dataset and the sign of the same coefficient turns around; it is shown side by side that the coefficient of determination does not announce this turning.