--- title: 'Visual Inspection of Data' source: 'https://academia.sh/en/courses/statistical-analysis-and-exploratory-data-analysis/visual-inspection-of-data' course: 'Statistical Analysis and Exploratory Data Analysis' language: en updated: '2026-10-07T04:31:13+00:00' license: 'CC BY-SA 4.0' objectives: - 'Catches heaping, truncation, impossible combinations and period gaps by counting them with threshold rules' - 'Measures which summary each record problem moves and by how much without deleting rows: two computations side by side and the difference' - 'Computes the number of breakdown cells opened by the exploration stage and defines the debt handed over to testing' --- # Visual Inspection of Data A six-rule scan finds four problems in the raw record layer and each one distorts a different summary: the value 0.00 repeats 119 times and holds the dataset mean at 19.693, whereas it would be 20.454 if the rows that do not count as readings were left out; 20 records have piled up at the 45.00 m³ ceiling and the mean of the largest 1% appears as 44.663 instead of 46.597; 213 rows carry measured consumption in a zero-person household and consumption per person comes out as 11.204 or 10.617 depending on the decision; the row range between periods is 33.5% and the first-period mean rises from 20.644 to 20.868 when balanced subscribers are taken. No row is deleted. At the close of the topic the debt opened by exploration is counted: five breakdown fields produce 23 single and 209 pair cells, 232 separate subsets in total, and 33 of them have fewer than 30 rows. The first four lessons of this topic always approached the dataset through computation: a mean was taken, a distribution was counted, a coefficient was computed, a matrix was built. Their common assumption was that the values in the record were measured values. Yet during exploration the first things caught are often not found by computation but seen while looking at the table: the repetition of the same value with suspicious frequency, records piling up at a limit, field combinations that should never occur together, one period being markedly sparser than the others. The unit of measure of this lesson is **the number of problems the inspection catches and the summary each one distorts**. Four problems are found, a single question is asked for each — which number does it move and by how much — and no row is deleted. - **EA21.** Visual inspection here is **not drawing** but a scan turned into numbers: how many times each value is repeated, where records pile up, which field combination cannot exist, which slice is sparse. - **EA22. Heaping** is the repetition of the same value more than expected; its measure is the row share of the most frequent value. The same number appearing hundreds of times in a continuous quantity is not a measurement but a recording convention. - **EA23. Truncation** is the piling up of records at an upper or lower limit; its measure is the number of records at that limit. Truncation **cannot be repaired**: the truncated value is not in the record. The only things that can be counted are how many records were truncated and which summary was distorted by how much. - **EA24. An impossible combination** is two individually valid fields being invalid together; its measure is the number of rows that break the rule. - **EA25.** In this lesson no row is **deleted**. Cleaning was the job of the Data Collection and Preparation course and the measure there was the number of rows; here, for each problem two computations are printed side by side and the difference between them is written. ## Six Rules, Four Findings ```python # exploration.py — MODEL: the fictional measurement network of K03 and K04 is rebuilt # from the same seed. The dataset is fictional; no real subscriber, meter or region appears. import math from collections import Counter SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] TARIFFS = [10, 25, 40] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t: int): """Linear congruential generator: seed written in the code, same output every run.""" x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "tier": 1 if t < TARIFFS[0] else 2 if t < TARIFFS[1] else 3 if t < TARIFFS[2] else 4}) r() def mean(v: list[float]) -> float: return sum(v) / len(v) def align(w: list[int], left: int = 1): """The first `left` columns are left-aligned, the rest right-aligned.""" return lambda h: "".join(str(v).ljust(w[i]) if i < left else str(v).rjust(w[i]) for i, v in enumerate(h)) # The raw record layer is a MODEL: the field device cannot read above 45.00 m3 and writes # the ceiling. This is the table that reaches the analyst; READINGS exists only for comparison. CEILING = 45.0 RECORDS = [dict(o, consumption_m3=min(o["consumption_m3"], CEILING)) for o in READINGS] T = [k["consumption_m3"] for k in RECORDS] top = Counter(T).most_common(1)[0] period_counts = Counter(k["period"] for k in RECORDS).values() duplicates = Counter((k["subscriber_id"], k["period"]) for k in RECORDS).values() RULES = [ ("heaping", "row share of the most frequent value (percent)", 100 * top[1] / len(T), 1.0, f"value {top[0]:.2f} repeated {top[1]} times"), ("truncation", "records at the largest value", float(Counter(T)[max(T)]), 5.0, f"{Counter(T)[max(T)]} records piled up at the {max(T):.2f} m3 upper limit"), ("impossible combination", "rows with consumption > 0 when household is 0", float(sum(1 for k in RECORDS if k["household_size"] == 0 and k["consumption_m3"] > 0)), 1.0, "measured consumption in a zero-person household"), ("period gap", "row range between periods", 100 * (max(period_counts) - min(period_counts)) / max(period_counts), 10.0, f"sparsest period {min(period_counts)} rows, densest {max(period_counts)} rows"), ("duplicate record", "rows of the same subscriber-period pair", float(max(duplicates)), 1.5, "-"), ("undefined value", "rows with negative consumption", float(sum(1 for t in T if t < 0)), 1.0, "-"), ] d = align([24, 46, 9, 10], 2) print(d(["problem", "measured", "value", "threshold"]) + " finding") problems = 0 for name, measure, value, threshold, text in RULES: hit = value >= threshold problems += hit print(d([name, measure, f"{value:.1f}", f"{threshold:.1f}"]) + " " + (text if hit else "-")) print(f"\n{len(RULES)} rules run, {problems} problems found; none deletes a row") ``` ``` problem measured value threshold finding heaping row share of the most frequent value (percent) 3.7 1.0 value 0.00 repeated 119 times truncation records at the largest value 20.0 5.0 20 records piled up at the 45.00 m3 upper limit impossible combination rows with consumption > 0 when household is 0 213.0 1.0 measured consumption in a zero-person household period gap row range between periods 33.5 10.0 sparsest period 838 rows, densest 1260 rows duplicate record rows of the same subscriber-period pair 1.0 1.5 - undefined value rows with negative consumption 0.0 1.0 - 6 rules run, 4 problems found; none deletes a row ``` The record layer is a model and what it models is clear: the field device cannot read above 45.00 m³ and writes the ceiling. This is the table that reaches the analyst; the untruncated values are kept only for comparison. The four findings are of four different kinds. The value 0.00 appears in 119 of 3,199 rows, that is 3.7%; the repetition of a number at this frequency in a continuous quantity does not come from measurement but from a recording convention that writes zero for a meter that could not be read. At the largest value 20 records have piled up: the exact same largest value appearing twenty times in a quantity with two decimals is the signature of the ceiling. 213 rows carry measured consumption in a zero-person household; the `household_size` field is valid on its own, the `consumption_m3` field is valid on its own, their combination is invalid. The row range between periods is 33.5%. The two rules that did not exceed the threshold give information too. The same subscriber and period pair does not appear more than twice in any row and there is no row that writes negative consumption. These are problems that were searched for and not found, and the fact that they were searched for is written down; when it is not written, the reader assumes the inspection never asked that question. ## The Number Each Problem Moves ```python # effect.py — the same question for each problem: which summary does it move and by how much. No # row is deleted; two computations are printed side by side and the difference is written. def per_person(records, household_zero_is_one: bool) -> float: v = [k["consumption_m3"] / (max(k["household_size"], 1) if household_zero_is_one else k["household_size"]) for k in records if household_zero_is_one or k["household_size"] > 0] return mean(v) def period_mean(records, period: str, subscribers=None) -> float: return mean([k["consumption_m3"] for k in records if k["period"] == period and (subscribers is None or k["subscriber_id"] in subscribers)]) complete = {a for a, c in Counter(k["subscriber_id"] for k in RECORDS).items() if c == len(PERIODS)} TOP = round(0.01 * len(T)) EFFECT = [ ("heaping", top[1], "dataset mean (m3)", mean(T), mean([t for t in T if t > 0]), "if rows where zero is written are not counted as readings"), ("truncation", Counter(T)[max(T)], "mean of the largest 1% (m3)", mean(sorted(T)[-TOP:]), mean(sorted(o["consumption_m3"] for o in READINGS)[-TOP:]), "had there been no ceiling (known only in the model)"), ("impossible combination", sum(1 for k in RECORDS if k["household_size"] == 0), "consumption per person (m3)", per_person(RECORDS, True), per_person(RECORDS, False), "household-0 rows left out"), ("period gap", len(RECORDS) - 3 * len(complete), "first-period mean (m3)", period_mean(RECORDS, PERIODS[0]), period_mean(RECORDS, PERIODS[0], complete), "only subscribers with all three periods"), ] e = align([24, 7]) print(e(["problem", "rows"]) + " " + "distorted summary".ljust(30) + "recorded".rjust(10) + "problem-free".rjust(13) + "diff".rjust(9)) for name, n, summary, recorded, clean, _ in EFFECT: print(e([name, n]) + " " + summary.ljust(30) + f"{recorded:10.3f}{clean:13.3f}{clean - recorded:+9.3f}") print() for name, _, _, recorded, clean, explanation in EFFECT: print(f"{name:<24}{explanation}; {100 * (clean - recorded) / recorded:+.1f} percent") ``` ``` problem rows distorted summary recorded problem-free diff heaping 119 dataset mean (m3) 19.693 20.454 +0.761 truncation 20 mean of the largest 1% (m3) 44.663 46.597 +1.934 impossible combination 222 consumption per person (m3) 11.204 10.617 -0.586 period gap 685 first-period mean (m3) 20.644 20.868 +0.224 heaping if rows where zero is written are not counted as readings; +3.9 percent truncation had there been no ceiling (known only in the model); +4.3 percent impossible combination household-0 rows left out; -5.2 percent period gap only subscribers with all three periods; +1.1 percent ``` Four problems distort four different summaries and none of them substitutes for another. The zero heaping holds the dataset mean down: instead of 19.693, it would be 20.454 if the rows that do not count as readings were left out, that is 3.9%. This number is inside every region mean that is reported too, because the zeros are distributed over all regions. The number that truncation moves is not the mean but the upper end. The dataset mean is almost unaffected by the ceiling, but the mean of the largest 1% stays at 44.663; without the ceiling it would be 46.597. This second number is known only in the model. In the field a truncated value is not in the record and cannot be brought back; the only thing that can be done is to write the sentence "20 of the 32 highest readings rest on the ceiling" next to the number. In a truncated column the largest value, every summary above the quartiles and every decision concerning the upper end rest on a clipped number, not a measured one. The impossible combination is different: here there are two defensible decisions and both produce a number. If a zero-person household is counted as 1, consumption per person is 11.204 m³; if those 222 rows are left out it comes out as 10.617 m³. The difference is 5.2%, and the data does not say which is right; what says it is how many people the household actually has, and that is not in the record either. The number that the period gap moves is the smallest but its direction is instructive. The first-period mean is 20.644 m³ in the record; when only the subscribers with all three periods are taken it becomes 20.868 m³. The 0.224 m³ between them is not a behavior difference but a **composition** difference: the subscribers who drop out early consumed below the average in the first period. A sentence tying the decline between periods to behavior cannot be written without taking this difference into account. ## The Debt Exploration Opens ```python # debt.py — the number of breakdowns opened in the exploration stage: each cell is a subset that # can be looked at, and each look is a debt that enlarges the margin of the test. FIELDS = [("region", 5), ("period", 3), ("tier", 4), ("satisfaction", 5), ("household_size", 6)] single = sum(k for _, k in FIELDS) field_pairs = [(i, j) for i in range(len(FIELDS)) for j in range(i + 1, len(FIELDS))] double = sum(FIELDS[i][1] * FIELDS[j][1] for i, j in field_pairs) small, empty = 0, 0 for i, j in field_pairs: a, b = FIELDS[i][0], FIELDS[j][0] count = Counter((k[a], k[b]) for k in RECORDS) empty += FIELDS[i][1] * FIELDS[j][1] - len(count) small += sum(1 for v in count.values() if v < 30) print(f"{len(FIELDS)} breakdown fields, {len(field_pairs)} field pairs") print(f"single breakdown cells : {single}") print(f"double breakdown cells : {double}") print(f"total viewable subsets: {single + double}") print(f"{small} of them have fewer than 30 rows, {empty} are empty") print(f"the dataset is 1 table; exploration can extract {single + double} separate summaries") ``` ``` 5 breakdown fields, 10 field pairs single breakdown cells : 23 double breakdown cells : 209 total viewable subsets: 232 33 of them have fewer than 30 rows, 2 are empty the dataset is 1 table; exploration can extract 232 separate summaries ``` Five breakdown fields open 23 cells on their own, and their pair combinations 209 cells more. In total 232 subsets, all coming out of the same table, and for each of them a mean, a rate or a difference can be computed. 33 of them have fewer than 30 rows, and in a small cell the margin grows: the mean of a subset of twenty rows moves noticeably when a single record comes or goes. This number is not a defect but the nature of exploration. The exploration stage exists precisely to walk through these cells, look at which one draws attention and produce questions from there. But every cell walked is a debt added to the margin of the test to be done later — because the question to be tested was selected **because it was the one that drew the most attention** among the 232 cells. ## Summary - The six-rule scan found four problems: heaping (the value 0.00 in 3.7% of the rows), truncation (20 records at 45.00 m³), impossible combination (213 rows) and period gap (33.5%). - Heaping holds the dataset mean at 19.693; if the zeros are not counted as readings it becomes 20.454, so a range of 3.9% is inside every region mean. - Truncation hardly moves the mean at all but shows the mean of the largest 1% as 44.663 instead of 46.597 and **cannot be repaired**; the only thing that can be counted is how many records rest on the ceiling. - The impossible combination leaves two defensible decisions: consumption per person is 11.204 or 10.617 m³, a difference of 5.2%. - The period gap carries the first-period mean from 20.644 to 20.868; this is not a behavior but a composition difference. ## Next Step In five lessons this topic did one thing: it got to know the dataset and produced questions. It was seen that the 4.463 m³ difference between regions could not be produced by permutation, it was measured that the east distribution is skewed and that the 30 rows outside the fences that were not dropped moved the mean by 0.480 m³, it was shown that three different relationships giving the same coefficient separate by their slice means, it was counted that 12 of the 190 pairs in twenty independent columns exceed the threshold, and it was written that four record problems distorted four different summaries. The common gap is this: **none of these questions was tested**. Exploration says that a difference seems to exist, it does not say whether that difference is worth reporting; it says a summary is skewed, it does not decide which summary will be reported. Moreover, exploration itself left a debt: 232 breakdown cells were walked and the question to be tested was selected from among them. The next topic, Applied Statistics, begins by paying this debt — first it measures the choice of the summary to be reported and the number that choice moves, then it builds the difference's own interval and test.