Interpreting Confidence Intervals

In the previous two lessons intervals were produced twice and widths were written twice, but what an interval says was not defined. The confidence interval is not new in this curriculum: it was used as a tool in the Measurement topic of the User Experience and Behavior Design course, and the sample size and the multiple comparison correction were computed there. Those computations are not repeated here. In this lesson the interval itself is the subject.

Here the interval is produced not with a formula but with bootstrap. This choice is not a convenience: the formula route asks for an assumption about the distribution, and that assumption is disputable for reading data, while the draw route asks only for the dataset itself. The measure is gathered under two headings: what the width depends on and what the interval does not say. The most expensive item of the second heading is left for the end — the overlap of two intervals does not mean that two means are not different, and the proof of this is given with a number.

• AS12. A confidence interval is not a measurement but the output of a procedure. The dataset is drawn from 1,000 times at the same size and with replacement, the summary is recomputed at every draw, and the middle 95% of the resulting values is written as the interval.
• AS13. The interval belongs to the summary, not to the observations. It says nothing about where the individual readings stand, and the vast majority of the readings are outside the interval.
• AS14. The width depends on the observation count, and when the observation count is quadrupled the width falls to about half. The price of halving the margin of error is four times the observations.
• AS15. The 95% belongs not to a single interval but to the procedure. About 95% of the intervals built with the same procedure contain the population value; this is what can be said for the single interval at hand.
• AS16. The interval assumes that the dataset represents the population. It neither measures nor corrects selection bias; in a biased dataset a narrow interval is a narrow interval around the wrong number.
• AS17. The overlap of two intervals does not say “no difference”. The difference is a separate number.
• AS18. The place where the decision is read is the interval of the difference itself: it is checked whether zero is inside.

How the Interval Is Produced

# dataset.py — MODEL: the measurement network of K03-K04 is regenerated with the same seed.
# A long-format reading table of 3,199 rows, 1,329 subscribers, three periods, five regions.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
PERIODS = ["2026-01", "2026-02", "2026-03"]
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_kept, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (_dropped if r() < 0.046 else _kept).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {a["subscriber_id"] for a in _kept}

READINGS = []
for a in _kept + _dropped:
    r = rng(SEED + 7000 + a["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100
        if a["subscriber_id"] not in VALID:
            r()
            continue
        READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"],
                         "period": PERIODS[d], "consumption_m3": t})
        r()


def mean(v):
    return sum(v) / len(v)


DRAWS = 1000


def percentile(s, p):
    i = p * (len(s) - 1)
    a = int(i)
    z = min(a + 1, len(s) - 1)
    return s[a] + (s[z] - s[a]) * (i - a)


def interval(v, seed, draws=DRAWS):
    rand = rng(seed)
    n = len(v)
    d = sorted(mean([v[int(rand() * n)] for _ in range(n)]) for _ in range(draws))
    return percentile(d, 0.025), percentile(d, 0.975)


east = [o["consumption_m3"] for o in READINGS if o["region"] == "east"]
a, z = interval(east, SEED + 301)
print(f"east mean {mean(east):.3f} m3, n={len(east)}, {DRAWS} draws")
print(f"95% confidence interval {a:.3f} - {z:.3f} m3, width {z - a:.3f} m3")
inside = sum(1 for x in east if a <= x <= z)
print(f"{inside} of the 557 readings ({100 * inside / len(east):.2f} percent) are inside this interval")

east mean 24.674 m3, n=557, 1000 draws
95% confidence interval 23.894 - 25.431 m3, width 1.537 m3
37 of the 557 readings (6.64 percent) are inside this interval

The procedure is a single sentence: since there is no population at hand, the dataset is put in the place of the population, 1,000 draws of the same size are made from it with replacement, the mean is recomputed at every draw, and the middle 95% of the resulting 1,000 means is written as the interval. For the east region the result is 23.894–25.431 m³, and the margin of error, that is, the half-width of the interval, is 0.769 m³.

The last line turns the most commonly confused side of the interval into a number. The interval contains only 37 of the 557 readings, that is, 6.64%. The interval reports not where the readings are but where the mean may be. The sentence “95% of the subscribers consume between 23.9 and 25.4 m³” cannot be derived from this output and is false in the dataset.

In report language the distinction is kept like this: the interval is always written under the name of a summary — the interval of the mean, the notch of the median, the interval of the ratio. When the name is not written, the reader taking it for an interval of the observations is an expected misunderstanding.

The only assumption of the procedure also stands in the open: the dataset is accepted as representing the population. Bootstrap puts the dataset at hand in the place of the population that is not at hand, and the price of this substitution grows in small datasets. A value that is not in the dataset cannot appear in any draw, so the interval sees only as much as the dataset can see.

What the Width Depends On

# width.py — the readings of the same region are first shuffled, then three nested
# subsets are taken: 35, 139, 557. At every step the observation count is roughly quadrupled.
rand = rng(SEED + 304)
shuffled = east[:]
for i in range(len(shuffled) - 1, 0, -1):
    j = int(rand() * (i + 1))
    shuffled[i], shuffled[j] = shuffled[j], shuffled[i]
previous = None
for n in (35, 139, 557):
    sub = shuffled[:n]
    a2, z2 = interval(sub, SEED + 305)
    g = z2 - a2
    extra = "" if previous is None else f", {g / previous:.3f} times the previous width"
    print(f"n={n:3d} mean {mean(sub):6.3f} interval {a2:6.3f} - {z2:6.3f} "
          f"width {g:.3f}{extra}")
    previous = g

n= 35 mean 24.242 interval 20.825 - 27.426 width 6.601
n=139 mean 24.115 interval 22.507 - 25.515 width 3.009, 0.456 times the previous width
n=557 mean 24.674 interval 23.985 - 25.445 width 1.459, 0.485 times the previous width

The three subsets are nested: the 35 readings are inside the 139, and the 139 inside the 557. The observation count is roughly quadrupled at every step, and the width falls to roughly half at every step — from 6.601 to 3.009 and then to 1.459 m³. The ratios come out as 0.456 and 0.485, that is, around halving.

The counterpart of this in planning is direct: a measurement that wants to halve the margin of error must collect four times the observations. In a breakdown of thirty-five readings the mean has an interval 6.6 m³ wide, and in that breakdown nothing can be said about the 25 m³ tariff step; the observation count needed to be able to answer the same question is sixteen times as large. The price of cutting a breakdown finer is also visible here: every split lowers the observation count, and the falling observation count grows the width in inverse proportion to the square root.

It should be noticed that the means move too: 24.242, 24.115 and 24.674. All three numbers are correct and all three come from the same region. The difference among them is exactly what the interval measures.

The second determinant of the width is the dataset’s own dispersion. At the same observation count, a breakdown whose values stand close to each other gives a narrow interval, and a widely dispersed breakdown gives a wide one. The analyst has only one of these two levers: the observation count is the subject of the measurement plan, while dispersion is a property of the thing measured. This is why a narrow interval is not an achievement of analysis but mostly the result of a collection decision.

The 95% Is the 95% of What

# coverage.py — the 557 readings are accepted as the population, samples of 40 observations
# are drawn from it and a short interval is built for every sample. What is counted is
# how many of the intervals contain the population mean.
POPULATION = mean(east)
rand = rng(SEED + 308)
TRIALS, SAMPLE, SHORT = 120, 40, 200
covering = 0
for _ in range(TRIALS):
    sample = [east[int(rand() * len(east))] for _ in range(SAMPLE)]
    d = sorted(mean([sample[int(rand() * SAMPLE)] for _ in range(SAMPLE)])
               for _ in range(SHORT))
    if percentile(d, 0.025) <= POPULATION <= percentile(d, 0.975):
        covering += 1
print(f"population mean {POPULATION:.3f} m3 accepted; {TRIALS} samplings of "
      f"{SAMPLE} observations, interval built with {SHORT} draws")
print(f"{covering} of the intervals ({100 * covering / TRIALS:.1f} percent) contain the "
      f"population mean")

population mean 24.674 m3 accepted; 120 samplings of 40 observations, interval built with 200 draws
113 of the intervals (94.2 percent) contain the population mean

The number 95 does not belong to the single interval at hand; it belongs to the procedure. To see this, the 557 readings of the east region were taken as the population, 120 separate samples of 40 observations were drawn from it, and an interval was built for each. Of the 120 intervals built, 113 contain the population mean, that is, 94.2%.

This is the correct form of the sentence: about 95% of the intervals built with this procedure contain the population value. For the single interval at hand it cannot be said that “the population mean is inside with 95% probability”, because the population mean is a number and it is either inside or not; what moves is the interval, not the population value.

That the resulting number is 94.2 instead of 95 is also a piece of information and not a deviation to be corrected. In samples of forty observations the bootstrap builds the interval a little narrow; this ratio, measured with 120 trials, has a margin of its own of about two points. If the run count were enlarged the ratio would approach 95, but the speed of approach also obeys the very rule this lesson measures.

What the interval does not say also stays outside this run. If the dataset does not represent the population — if the network has only the meters that could be read — the interval neither sees nor corrects this deficiency; a narrow interval is only narrow around the wrong number. Selection bias is the job of the last lesson of this topic.

The Overlap of Two Intervals

# overlap.py — the two periods of north: first two separate intervals, then the interval
# of the difference itself. The difference is recomputed at every draw by drawing
# separately from the two groups.
north2 = [o["consumption_m3"] for o in READINGS
          if o["region"] == "north" and o["period"] == "2026-02"]
north3 = [o["consumption_m3"] for o in READINGS
          if o["region"] == "north" and o["period"] == "2026-03"]
a2, z2 = interval(north2, SEED + 306)
a3, z3 = interval(north3, SEED + 306)
print(f"north 2026-02: n={len(north2)} mean {mean(north2):.3f} interval {a2:.3f} - {z2:.3f}")
print(f"north 2026-03: n={len(north3)} mean {mean(north3):.3f} interval {a3:.3f} - {z3:.3f}")
print(f"the two intervals overlap for {min(z2, z3) - max(a2, a3):.3f} m3")
rand = rng(SEED + 307)
n2, n3 = len(north2), len(north3)
f = sorted(mean([north2[int(rand() * n2)] for _ in range(n2)])
           - mean([north3[int(rand() * n3)] for _ in range(n3)])
           for _ in range(DRAWS))
fa, fz = percentile(f, 0.025), percentile(f, 0.975)
print(f"interval of the difference itself: {mean(north2) - mean(north3):.3f} m3 "
      f"({fa:.3f} - {fz:.3f}), is zero inside: {fa <= 0 <= fz}")

north 2026-02: n=319 mean 20.324 interval 19.491 - 21.193
north 2026-03: n=239 mean 18.879 interval 18.068 - 19.639
the two intervals overlap for 0.148 m3
interval of the difference itself: 1.446 m3 (0.281 - 2.612), is zero inside: False

The two periods of the north region were put side by side. The two intervals overlap by 0.148 m³, and a reader who looks at the overlap draws the conclusion “there is no difference between the two periods”. This conclusion is wrong, and its wrongness is not an interpretive preference but something that can be shown by computation.

When the interval of the difference itself is computed, the observed difference of 1.446 m³ settles into the 0.281–2.612 m³ interval, and zero stays outside this interval. That is, the same data says “uncertain” when looking at the two separate intervals and “away from zero” when looking at the interval of the difference. Of the two, the correct one is the second, because the question asked is what the difference is, and the difference is a separate number.

The reason in short is this: two separate intervals put two separate uncertainties side by side at their full length, whereas the uncertainty of the difference is not as large as the sum of the two. Non-overlap always shows that the difference is away from zero, but overlap shows nothing. If a decision is to be made, the number to look at is single: the interval of the difference and the place of zero in that interval.

The rule in practice follows from this. If the intervals of two groups are printed side by side in a report, that is not the place where the comparison is made; the interval of the difference is computed separately and written separately. A two-interval display is useful for introducing each group on its own, not for comparison. When this distinction is not written, the same output produces two different decisions, and which one is given depends on which line the reader looks at.

Summary

• The confidence interval was produced not with a formula but with bootstrap: the east region’s mean of 24.674 m³ settled into the 23.894–25.431 m³ interval in 1,000 draws, that is, into a margin of error of 0.769 m³.
• The interval belongs to the summary, not to the observations; only 37 of the 557 readings, 6.64%, are inside this interval.
• When the observation count rose from 35 to 139 and from 139 to 557, the width fell from 6.601 to 3.009 and then to 1.459 m³; the price of halving the margin of error is four times the observations.
• The 95% belongs not to a single interval but to the procedure: in 113 of 120 samples of forty observations, that is, in 94.2%, the interval contained the population mean.
• The two periods of north gave two intervals that overlap by 0.148 m³, but the interval of the difference itself came out as 0.281–2.612 m³ and left zero outside; overlap does not mean no difference.

Next Step

The interval of the difference left zero outside, and this is the output that stands closest to a decision. Still, it is not a test by itself: the question not asked is “how often a difference of this size would appear when there is no difference at all”. The next lesson answers this question with a permutation test, defines the null hypothesis, the p-value and the significance level through permutation, and tests the same two groups by counting them both as paired and as independent. The p-value is printed twice, the effect size stands next to it, and what the p-value is not is shown with numbers.