Applying Hypothesis Testing

The previous lesson computed the interval of the difference itself and saw that zero stayed outside the interval. That stood close to a decision but it was not a test. The question a test asks is different: when there is no difference at all, in how many runs out of how many does a difference of the observed size appear.

In this course the answer to that question is not read from a table; it is counted by running. The first lesson of the exploratory analysis topic randomly redistributed the region labels and counted in how many runs the difference between east and north appeared; here the same method is named a test, and the main measure is this: the choice of test is an assumption, and when the assumption changes the p-value moves. The same two groups, the same observed difference, two p-values.

• AS19. The null hypothesis is not a sentence but a procedure. The assumption “there is no difference between the two groups” means that the labels are interchangeable among the observations; the permutation test does exactly this and recomputes the difference at every shuffle.
• AS20. The p-value is how many of the shuffled datasets produced a difference as large as the observed one or larger. The observed run also takes part in the count, so the numerator and the denominator are each increased by one.
• AS21. The significance level is a threshold written before the test. A threshold chosen after the result is seen is not a threshold but a result.
• AS22. The choice of test is an assumption, and the assumption can be checked. Whether the pairing is real is measured with the correlation of the paired readings.
• AS23. The same observed difference gives two p-values under two assumptions. The difference does not change, the p changes.
• AS24. The p-value is not the size of the effect. The effect size, the difference divided by the standard deviation, is written next to p as a separate number.
• AS25. The p-value is not the probability of the null hypothesis either. Moreover its resolution depends on the run count: the smallest p-value that can be written with 2,000 permutations is 0.00050.

The Null Hypothesis Is a Procedure

# dataset.py — MODEL: the fictional measurement network of K03-K04 is regenerated with the same seed.
# A long-format reading table of 3,199 rows, 1,329 subscribers, three periods, five regions.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
PERIODS = ["2026-01", "2026-02", "2026-03"]
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_kept, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (_dropped if r() < 0.046 else _kept).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {a["subscriber_id"] for a in _kept}

READINGS = []
for a in _kept + _dropped:
    r = rng(SEED + 7000 + a["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100
        if a["subscriber_id"] not in VALID:
            r()
            continue
        READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"],
                         "period": PERIODS[d], "consumption_m3": t})
        r()


def mean(v):
    return sum(v) / len(v)


PERMUTATIONS = 2000
by_subscriber = {}
for o in READINGS:
    by_subscriber.setdefault(o["subscriber_id"], {})[o["period"]] = o["consumption_m3"]
PAIRS = [(v["2026-01"], v["2026-02"]) for v in by_subscriber.values()
         if "2026-01" in v and "2026-02" in v]
first = [x for x, _ in PAIRS]
second = [y for _, y in PAIRS]
diff = [x - y for x, y in PAIRS]
OBSERVED = mean(diff)
print(f"subscribers read in both periods: {len(PAIRS)}")
print(f"2026-01 mean {mean(first):.3f} m3, 2026-02 mean {mean(second):.3f} m3")
print(f"observed difference {OBSERVED:.3f} m3")


def shuffle(v, draw):
    h = v[:]
    for i in range(len(h) - 1, 0, -1):
        j = int(draw() * (i + 1))
        h[i], h[j] = h[j], h[i]
    return h


draw = rng(SEED + 401)
everything, n1 = first + second, len(first)
exceeded = 0
for _ in range(PERMUTATIONS):
    h = shuffle(everything, draw)
    if abs(mean(h[:n1]) - mean(h[n1:])) >= abs(OBSERVED):
        exceeded += 1
p_independent = (exceeded + 1) / (PERMUTATIONS + 1)
print(f"independent assumption: {exceeded} of {PERMUTATIONS} permutations exceeded the observed difference, "
      f"p = {p_independent:.5f}")

subscribers read in both periods: 1101
2026-01 mean 20.638 m3, 2026-02 mean 19.681 m3
observed difference 0.957 m3
independent assumption: 24 of 2000 permutations exceeded the observed difference, p = 0.01249

The question is a report question: did consumption fall from the first period to the second. There are 1,101 subscribers who could be read in both periods, the means are 20.638 and 19.681 m³, and the observed difference is 0.957 m³. The dataset is fictional and regenerated with the same seed from the measurement network of K03–K04; that only 1,101 of the 1,329 subscribers could be read in both periods is the dataset’s own deficiency, not a selection.

The null hypothesis is set up here not as a sentence but as a procedure. Saying “the period label has no effect” means accepting that which period each of the 2,202 readings belongs to is random. The code does exactly this: it puts all the readings in a single bag, shuffles it, counts the first 1,101 as the first period, and recomputes the difference. This is repeated 2,000 times.

The p-value is a count from this run: 24 of the permutations produced a difference as large as the observed 0.957 m³ or larger. One is added to the numerator and the denominator, because the observed arrangement is itself one of the possible arrangements; p comes out as 0.01249. If a significance level of 5% was written before the test, this result is below the threshold.

That the significance level is written before is the load-bearing part of this sentence. When the threshold is chosen after the result, it stops being a decision rule and turns into a label that repeats the result itself; in that case the test has eliminated nothing. The number of the threshold is also a convention, not a measurement: in this dataset the choice between 5% and 1% does more work than which of the two assumptions below is looked at.

The Pairing Assumption

# paired.py — the same dataset, another assumption: every row is the two readings of one subscriber.
# This time the null hypothesis means "within every pair the sign of the difference is random".
draw = rng(SEED + 403)
exceeded2 = 0
for _ in range(PERMUTATIONS):
    t = sum(f if draw() < 0.5 else -f for f in diff) / len(diff)
    if abs(t) >= abs(OBSERVED):
        exceeded2 += 1
p_paired = (exceeded2 + 1) / (PERMUTATIONS + 1)


def std_dev(v):
    m = mean(v)
    return math.sqrt(sum((x - m) ** 2 for x in v) / (len(v) - 1))


pooled = math.sqrt((std_dev(first) ** 2 + std_dev(second) ** 2) / 2)
ma, mb = mean(first), mean(second)
cov = sum((x - ma) * (y - mb) for x, y in PAIRS) / (len(PAIRS) - 1)
print(f"paired assumption: {exceeded2} permutations exceeded the observed difference, p = {p_paired:.5f}")
print(f"assumption check: correlation of the paired readings {cov / (std_dev(first) * std_dev(second)):.3f}")
print(f"the difference is {OBSERVED:.3f} m3 in both; effect size "
      f"{OBSERVED / pooled:.3f} (independent) and {OBSERVED / std_dev(diff):.3f} (paired)")
print(f"p ratio {p_independent / p_paired:.2f} times")

paired assumption: 5 permutations exceeded the observed difference, p = 0.00300
assumption check: correlation of the paired readings 0.213
the difference is 0.957 m3 in both; effect size 0.112 (independent) and 0.089 (paired)
p ratio 4.17 times

The same 2,202 readings were counted with a second assumption. This time every row is the two readings of one subscriber, and the null hypothesis takes the form “within every pair the sign of the difference is random”. The shuffle changes accordingly: the labels swap places not between the groups but within every pair. The result is p 0.00300.

The observed difference is 0.957 m³ under both assumptions and has not moved by a single cubic meter. The only number that moved is the p-value, and it shrank 4.17 times. The whole difference between them comes from the assumption taken while choosing the test. Both assumptions stay below the 5% threshold, but at the 1% threshold the decisions part ways.

The assumption check is in the second line. For the pairing to bring a gain, the two readings of the same subscriber must be related to each other; the measured correlation is 0.213. This means that the subscribers’ consumption level is partly preserved across periods, and it explains why the paired assumption gave a smaller p: the dispersion between subscribers is subtracted from the computation.

The effect size stands next to p on the same line: 0.112 and 0.089. Both are the difference divided by one standard deviation, and both are small. What is being talked about is a decline of 4.6% of mean consumption; the smallness of the p-value does not enlarge this decline.

Pairing Does Not Come from the Procedure

# fake.py — pairing is not a procedure but a fact: if the second-period readings
# are matched with other subscribers the difference stays the same, the gain disappears.
draw = rng(SEED + 404)
fake_second = shuffle(second, draw)
fake_diff = [x - y for x, y in zip(first, fake_second)]
draw = rng(SEED + 405)
exceeded3 = 0
for _ in range(PERMUTATIONS):
    t = sum(f if draw() < 0.5 else -f for f in fake_diff) / len(fake_diff)
    if abs(t) >= abs(mean(fake_diff)):
        exceeded3 += 1
print(f"fake pairing: same difference {mean(fake_diff):.3f} m3, "
      f"p = {(exceeded3 + 1) / (PERMUTATIONS + 1):.5f}")
print(f"standard deviation of the differences: with the real pairing {std_dev(diff):.3f}, "
      f"with the fake {std_dev(fake_diff):.3f} m3")

fake pairing: same difference 0.957 m3, p = 0.01099
standard deviation of the differences: with the real pairing 10.753, with the fake 12.275 m3

It can be thought that the small p-value of the paired test was produced by the test itself. This run tests that: the second-period readings are shuffled and matched with other subscribers, and then exactly the same paired test is applied.

The observed difference does not change, because the mean of the two groups has stayed the same: again 0.957 m³. But p rises to 0.01099 and returns to the same region as the 0.01249 that the independent count gave. The reason is in the last line: the standard deviation of the differences has risen from 10.753 to 12.275 m³, that is, the dispersion the real pairing had lowered has come back.

The result gives the rule about the choice of test. The paired test is not a more powerful method; if the pairing is real it is more powerful. Inventing a pairing that does not exist does not shrink the p-value, it only moves the computation to the wrong place. This is why the assumption check is written not as an appendix to the test but as its precondition. The check itself is not expensive either: here a single correlation number and a single standard deviation comparison were enough, and both are too small to be counted next to the cost of the test.

What the p-Value Is Not

# scale.py — the same effect, a different observation count. The subsets are chosen
# nested and the same paired test is applied to each.
draw = rng(SEED + 406)
order = shuffle(list(range(len(PAIRS))), draw)
for divisor in (4, 2, 1):
    sub = [diff[i] for i in sorted(order[:len(PAIRS) // divisor])]
    r2 = rng(SEED + 403)
    over4 = 0
    for _ in range(PERMUTATIONS):
        t = sum(f if r2() < 0.5 else -f for f in sub) / len(sub)
        if abs(t) >= abs(mean(sub)):
            over4 += 1
    print(f"n={len(sub):4d} difference {mean(sub):6.3f} m3 effect {mean(sub) / std_dev(sub):.3f} "
          f"p = {(over4 + 1) / (PERMUTATIONS + 1):.5f}")
print(f"smallest p that can be written with {PERMUTATIONS} permutations = {1 / (PERMUTATIONS + 1):.5f}")

n= 275 difference  1.132 m3 effect 0.099 p = 0.10695
n= 550 difference  1.192 m3 effect 0.110 p = 0.01149
n=1101 difference  0.957 m3 effect 0.089 p = 0.00300
smallest p that can be written with 2000 permutations = 0.00050

The three lines show directly what the p-value is not. In the subset of a quarter the observed difference is 1.132 m³ — larger than the 0.957 m³ of the full dataset — and the effect size, 0.099, is also above 0.089. Even so the p-value is 0.10695, that is, thirty-five times the full dataset’s 0.00300 and above the 5% threshold.

The same three lines are also read the other way: as p shrank, the effect did not grow. The line that gives the smallest p is the line that gives the smallest effect. The p-value measures not the size of the difference but how far the difference, together with the observation count, is from chance. This is why p is not reported alone; the observation count and the effect size are written next to it.

The second wrong reading closes here too: p is not the probability that the null hypothesis is true. What it measures is how often a difference this large is produced when the null hypothesis is true — the number 0.00300 does not mean “the chance of no difference is three in a thousand”, it means “when there is no difference, a result like this came up in 5 of 2,000 permutations”.

The last line writes the limit of the number itself: the smallest p that can be produced with 2,000 permutations is 0.00050. In a test where no shuffle could exceed the observed difference, what can be written is not “p is zero” but “p is smaller than 0.00050”. The run count sets the resolution of the margin.

Summary

• The null hypothesis in the permutation test is a procedure: the period labels are randomly redistributed 2,000 times, the difference is recomputed each time, and the runs that exceed the observed difference are counted.
• In the same 1,101 pairs the observed difference is 0.957 m³ and does not change; the p-value comes out as 0.01249 in the independent assumption and 0.00300 in the paired assumption, that is, the assumption moves p by 4.17 times.
• The pairing assumption is checked: the correlation of the paired readings is 0.213. When the pairing is broken the standard deviation of the differences rises from 10.753 to 12.275 m³ and p returns to the independent assumption with 0.01099.
• The p-value is not the effect size: in the subset of 275 pairs the difference is 1.132 m³ and the effect is larger at 0.099, yet p is 0.10695, thirty-five times that of the full dataset.
• The p-value is not the probability of the null hypothesis either, and its resolution depends on the run count; the smallest p that can be written with 2,000 permutations is 0.00050.

Next Step

In this lesson a single question was asked and a single p-value was written: is there a difference between the two periods. Once the question is chosen, the test itself is orderly work and, together with the assumption check, gives a defensible result. In a real analysis, however, the question is not single. Which metric will be counted from the same experiment dataset, which breakdown will be opened, which observation window will be taken and what will be done about an unbalanced assignment are chosen one by one; every choice means a new test, and the number of questions enlarges the margin. The next lesson counts how many different decisions come out of the same experiment dataset and measures how much of this margin pre-registration closes.