Purpose of Exploratory Analysis

The Data Analysis with Python course built a toolkit over twenty-two lessons and wrote down every default of it. Two numbers were left at the close: the mean consumption of the east region is 24.674 m³, that of the north 20.210 m³. The report wrote the difference as 4.464 m³ and stopped there. Two questions went unasked: do these numbers summarize all the meters in the network, or only the part that could be read, and is the 4.464 m³ difference a regional difference, or a fluctuation that chance can produce at the same size?

This lesson asks those two questions and sets the unit of measure of the course. From here on, every number has a second number next to it: the margin of variation, how much that value could move with nothing changing. The rule of the course follows from this: a difference written without its margin of variation is an observation, not a finding. A number whose margin is not written counts as unmeasured.

• EA1. The dataset is fictional and carried over from K03–K04: a long-format reading table of 3,199 rows, 1,329 subscribers, three periods, five regions, and tariff tiers of 10/25/40 m³. The seed is written in the code; the numbers are the same from run to run.
• EA2. Exploratory data analysis generates hypotheses; it does not confirm them. Its output is not a decision but a list of questions to be answered; confirmation is a separate job.
• EA3. This dataset is not a population but a sample. Every observed summary is a summary of the rows at hand, not of the population, and would have come out differently in another sample.
• EA4. The margin of variation is measured in this course with permutation: group labels are randomly redistributed, the same number is recomputed, and the number of runs in which the observed value is exceeded is counted. No distribution table or critical value is used.
• EA5. The resolution of the margin is limited by the run count. For a difference that is exceeded in none of 2,000 runs, what can be said is not “its margin is zero” but “its margin is below 1 in 2,000”.

Population or Sample

# exploration.py — MODEL: the fictional measurement network of K03 and K04 is rebuilt
# from the same seed. The dataset is fictional; no real subscriber, meter or region appears.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
PERIODS = ["2026-01", "2026-02", "2026-03"]
TARIFFS = [10, 25, 40]
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t: int):
    """Linear congruential generator: seed written in the code, same output every run."""
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value() -> float:
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u: float, w: list[float]) -> int:
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_subscribers, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2],
         "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
         "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1}
    (_dropped if r() < 0.046 else _subscribers).append(k)
_valid = {a["subscriber_id"] for a in _subscribers}

READINGS = []
for k in _subscribers + _dropped:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp(
            (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100
        if k["subscriber_id"] not in _valid:
            r()
            continue
        READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"],
                         "period": PERIODS[d], "consumption_m3": t,
                         "household_size": k["household_size"],
                         "satisfaction": k["satisfaction"],
                         "tier": 1 if t < TARIFFS[0] else 2 if t < TARIFFS[1]
                         else 3 if t < TARIFFS[2] else 4})
        r()


def mean(v: list[float]) -> float:
    return sum(v) / len(v)


def region_values(name: str) -> list[float]:
    return [o["consumption_m3"] for o in READINGS if o["region"] == name]


east, north = region_values("east"), region_values("north")
POSSIBLE = len(_valid) * len(PERIODS)
print(f"fictional dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers")
print(f"east  {len(east):4d} rows, mean {mean(east):.3f} m3")
print(f"north {len(north):4d} rows, mean {mean(north):.3f} m3")
print(f"observed difference {mean(east) - mean(north):.3f} m3")
print(f"meters in network {RAW}, subscribers in records {len(_valid)}, "
      f"never in records {RAW - len(_valid)}")
print(f"possible readings {POSSIBLE}, observed readings {len(READINGS)}, "
      f"missing {POSSIBLE - len(READINGS)}, "
      f"coverage {100 * len(READINGS) / POSSIBLE:.1f} percent")

fictional dataset (seed 20260218): 3199 rows, 1329 subscribers
east   557 rows, mean 24.674 m3
north  921 rows, mean 20.210 m3
observed difference 4.463 m3
meters in network 1400, subscribers in records 1329, never in records 71
possible readings 3987, observed readings 3199, missing 788, coverage 80.2 percent

The last two lines close the question. The network has 1,400 meters, and the number of subscribers that entered the records is 1,329; 71 meters appear in no row. Had the subscribers in the records been read over three periods, there would be 3,987 rows; the observed rows number 3,199 and coverage stays at 80.2%. This dataset is not the whole network but the part of it that could be read — that is, a sample, and not a randomly drawn one either, because what selects which meter could be read is not a lottery but the field itself.

The consequence is direct: 24.674 m³ is not the mean consumption of the east region but the mean of the 557 rows that could be read in the east region. 557 rows drawn from the same network in another week would have given another number. The question itself changes here: the question “is the number correct” is replaced by “how much does this number move”.

The close of the previous course wrote the difference as 4.464 m³, while the running block writes 4.463 m³: the close subtracted two rounded numbers, the block the exact values. The 0.001 m³ between them changes nothing here, but it is the smallest example of the fact that a reported number depends on how it was computed.

The Margin of Variation of a Difference

One way to ask whether a difference is real is to build a world in which the difference does not exist and count how often a difference of the same size appears there. If the region label says nothing about consumption, randomly redistributing the labels should not change the result.

# margin.py — margin of variation: region labels are randomly redistributed, the same
# difference is recomputed, and the runs that exceed the observed value are counted.
RUNS = 2000
TOLERANCE = 1e-9  # one difference computed two ways shifts in the last digit; equality is not tested


def permutation_margin(a: list[float], b: list[float], seed: int):
    """The values of both groups are pooled. In each run len(a) items of the pool
    are picked at random as the first group and the difference is measured again."""
    r = rng(seed)
    pool, na, nb = a + b, len(a), len(b)
    total, observed = sum(pool), mean(a) - mean(b)
    exceeded, extreme = 0, 0.0
    for _ in range(RUNS):
        h, s = pool[:], 0.0
        for i in range(na):
            j = i + int(r() * (na + nb - i))
            h[i], h[j] = h[j], h[i]
            s += h[i]
        diff = s / na - (total - s) / nb
        if abs(diff) >= abs(observed) - TOLERANCE:
            exceeded += 1
        extreme = max(extreme, abs(diff))
    return observed, exceeded, extreme


def north_half(odd: bool) -> list[float]:
    return [o["consumption_m3"] for o in READINGS
            if o["region"] == "north" and (o["subscriber_id"] % 2 == 1) == odd]


COMPARISONS = [("east / north", east, north, SEED + 501),
               ("north: odd / even subscriber id", north_half(True),
                north_half(False), SEED + 502)]
WIDTHS = [33, 7, 7, 12, 12, 10]
y = lambda h: "".join(str(v).rjust(WIDTHS[i]) if i else str(v).ljust(WIDTHS[0])
                      for i, v in enumerate(h))
print(y(["comparison", "n1", "n2", "observed", "extreme", "exceeded"]))
counts = {}
for name, a, b, seed in COMPARISONS:
    observed, exceeded, extreme = permutation_margin(a, b, seed)
    counts[name] = exceeded
    print(y([name, len(a), len(b), f"{observed:.3f}", f"{extreme:.3f}", exceeded]))
print(f"\n{RUNS} runs; exceeded = runs that produced a difference at least as large as the observed one")
for name, exceeded in counts.items():
    print(f"{name:<33}margin of variation " + (f"below 1/{RUNS}" if exceeded == 0
                                              else f"{exceeded / RUNS:.3f}, exceeded in "
                                                   f"{round(100 * exceeded / RUNS)} of 100 runs"))

comparison                            n1     n2    observed     extreme  exceeded
east / north                         557    921       4.463       1.754         0
north: odd / even subscriber id      452    469       0.232       1.647      1314

2000 runs; exceeded = runs that produced a difference at least as large as the observed one
east / north                     margin of variation below 1/2000
north: odd / even subscriber id  margin of variation 0.657, exceeded in 66 of 100 runs

Two rows went through the same procedure and gave opposite answers. None of the 2,000 permutations reached the 4.463 m³ difference between east and north; the most extreme difference is 1.754 m³, less than half of the observed one. The region label says something about consumption.

The second row is a check. Whether a subscriber ID is odd or even cannot affect consumption; this breakdown is noise by definition, and the tool says so: the observed difference of 0.232 m³ is exceeded in 1,314 of 2,000 runs, that is, in sixty-six of every hundred runs. Writing such a difference into the report as “odd-numbered subscribers in the north consume more” would mean counting as a finding something that permutation produces two thirds of the time.

The resolution limit shows in the first row. A run count of zero exceeded does not mean “the margin is zero”; with 2,000 runs the smallest measurable margin is 1 in 2,000, and the observed difference is below it. Had the run count been raised tenfold, the limit would have dropped tenfold. The margin is itself a measurement and is written together with its own resolution.

What the procedure does not say is also written. Permutation answers only the question “how often does a difference of this size appear when the labels are permuted”; it does not say the cause of the difference. The tariff base of the east may be higher, household size may differ, or the meters that could be read in the east may systematically be the large subscribers. A small margin does not take the place of an explanation.

Exploration Does Not Decide, It Produces Questions

Measuring the margin of a difference is a step of exploration, not its end. The job of exploration is to walk through the dataset and work out which questions are worth asking; the scan below writes this as a list of measures and turns every measure that crosses its threshold into a question.

# question.py — the output of exploration is a list of questions, not a decision. Each
# check computes a measure; one that crosses its threshold yields a question, not a decision.
from collections import Counter


def median(v: list[float]) -> float:
    s = sorted(v)
    n = len(s)
    return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2


T, S = [o["consumption_m3"] for o in READINGS], sorted(o["consumption_m3"] for o in READINGS)
region_count = Counter(o["region"] for o in READINGS)
period_count = Counter(o["period"] for o in READINGS)
tier_count = Counter(o["tier"] for o in READINGS)
subscriber_count = Counter(o["subscriber_id"] for o in READINGS)
satisfaction_count = Counter(o["satisfaction"] for o in READINGS)
share = lambda k: 100 * k / len(READINGS)

CHECKS = [
    ("consumption", "share of rows written as zero", share(sum(1 for t in T if t == 0)),
     1.0, "upper", "is zero a measurement or an unread meter"),
    ("coverage", "observed / possible readings", 100 * len(READINGS) / POSSIBLE,
     95.0, "lower", "do the read differ from the unread"),
    ("subscriber_id", "share of subscribers with 3 periods",
     100 * sum(1 for v in subscriber_count.values() if v == 3) / len(subscriber_count),
     90.0, "lower", "are the missing periods randomly distributed"),
    ("region", "smallest / largest region",
     100 * min(region_count.values()) / max(region_count.values()),
     60.0, "lower", "how much does the margin of variation of the small region grow"),
    ("region", "share of rows, empty region label",
     share(sum(1 for o in READINGS if not o["region"])), 0.0, "upper",
     "which region do the unlabeled rows belong to"),
    ("period", "row range between periods",
     100 * (max(period_count.values()) - min(period_count.values())) / max(period_count.values()),
     10.0, "upper", "is the period difference behavior or the calendar"),
    ("consumption", "mean - median range",
     100 * (mean(T) - median(T)) / median(T), 2.0, "upper",
     "is the distribution skewed, which summary is reported"),
    ("consumption", "largest value / third quartile", max(S) / S[int(0.75 * len(S))],
     2.5, "upper", "how much do extreme values move the mean"),
    ("tier", "share of the rarest tier", share(min(tier_count.values())),
     5.0, "lower", "how much does the ratio of the rare tier vary"),
    ("satisfaction", "share of most common satisfaction level",
     share(max(satisfaction_count.values())),
     30.0, "upper", "is a mean legitimate on an ordinal scale"),
]

E = [15, 42, 8, 11]
z = lambda h: "".join(str(v).ljust(E[i]) if i < 2 else str(v).rjust(E[i])
                      for i, v in enumerate(h[:4])) + "  " + str(h[4])
print(z(["field", "measure", "value", "threshold", "question produced"]))
questions = 0
for field, measure, value, threshold, direction, text in CHECKS:
    fired = value > threshold if direction == "upper" else value < threshold
    questions += fired
    print(z([field, measure, f"{value:.1f}", f"{threshold:.1f}", text if fired else "-"]))
print(f"\n{len(CHECKS)} checks run, {questions} questions produced, 0 decisions made")

field          measure                                      value  threshold  question produced
consumption    share of rows written as zero                  3.7        1.0  is zero a measurement or an unread meter
coverage       observed / possible readings                  80.2       95.0  do the read differ from the unread
subscriber_id  share of subscribers with 3 periods           66.5       90.0  are the missing periods randomly distributed
region         smallest / largest region                     51.7       60.0  how much does the margin of variation of the small region grow
region         share of rows, empty region label              0.0        0.0  -
period         row range between periods                     33.5       10.0  is the period difference behavior or the calendar
consumption    mean - median range                            3.9        2.0  is the distribution skewed, which summary is reported
consumption    largest value / third quartile                 2.1        2.5  -
tier           share of the rarest tier                       1.8        5.0  how much does the ratio of the rare tier vary
satisfaction   share of most common satisfaction level       33.4       30.0  is a mean legitimate on an ordinal scale

10 checks run, 8 questions produced, 0 decisions made

Ten checks ran and eight questions came out. There is not a single decision in the table: no row says “these rows should be dropped” or “this region consumes more”. That is the output of exploration, and it is exactly where exploration parts from confirmation. Confirmation takes one question and produces one answer for it; exploration walks through the dataset and produces a list of questions to be answered.

The two rows that do not cross the threshold carry information as well. There are no rows with an empty region label, because the region comes from the subscriber record and was matched in the Data Collection and Preparation course; the ratio of the largest value to the third quartile is 2.1, that is, there is no extreme break at the upper end. A check that passes is also a measurement result.

The thresholds are themselves decisions. Had 75% been written instead of the 95% coverage threshold, the second row would not have produced a question; had 5% been chosen instead of 2% for the mean–median range, the seventh row would have gone silent. The number eight in the table is not a property of the dataset but a number produced jointly by the dataset and the threshold list. That is why the thresholds are written in the code and visible in the table.

The cost of the eight questions is paid in the next topic. Every breakdown looked at during exploration is a debt that enlarges the margin of the test: looking at ten measures and testing the most striking one is not the same as looking at a single measure and testing it. This lesson only writes that the debt was opened.

Summary

• The dataset is a sample, not a population: 1,329 of the 1,400 meters entered the records, 3,199 of 3,987 possible readings were observed, and coverage is 80.2%.
• The observed difference between east and north is 4.463 m³; when the region labels are randomly redistributed 2,000 times, the most extreme difference stays at 1.754 m³ and no run reaches the observed one.
• When the same procedure is applied to the rows of the north with odd and even subscriber IDs, the observed difference of 0.232 m³ is exceeded in 1,314 of 2,000 runs; the tool reports noise as noise.
• The resolution of the margin is the run count: for a difference that is never exceeded in 2,000 runs, what can be said is not “its margin is zero” but “its margin is below 1 in 2,000”.
• The output of exploration is a question, not a decision: the scan of ten checks produced 8 questions and made 0 decisions, and this number is the joint product of the dataset and the threshold list.

Next Step

This lesson looked at the difference between two region means and wrote its margin next to it. The mean itself was not questioned: when “the mean of 557 rows is 24.674 m³” is written, nothing has been said about how those rows are distributed, and distributions that give the same mean can be very different from each other. The next lesson covers univariate analysis: two distributions that give the same mean are built, the five-number summary and bin counts are printed, and it is counted how the range between the mean and the median gives skewness away and how much the outliers that were not removed move the summary.