--- title: 'Visualization, Reporting and Analysis at Scale' source: 'https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale' language: en updated: '2026-10-07T04:31:13+00:00' license: 'CC BY-SA 4.0' lessons: 18 outcomes: - 'Chooses the display that suits the data and the question by the read-back error of the encoding channel' - 'Builds a dashboard and its metrics so that they support decisions and tests whether they can be gamed' - 'Carries a finding to its target audience and makes the report reproducible' - 'Counts which part of a report can be divided when the scale grows' --- # Visualization, Reporting and Analysis at Scale Measuring how much of a finding survives as it is carried to the reader: the difference between the number an encoding channel lets the reader read back and the number in the source, the shape distribution and categorical displays cannot give back from the summary, the trend relationship and time displays carry, how area and color channels enlarge and shrink a difference, how many series a display that leans on a single channel leaves indistinguishable, the number of metrics that can be read back from a dashboard, whether a metric can be gamed without moving what it measures, the step that reproduces a report, and the report resting on scale: map–reduce passes, the values a columnar layout makes you read, and the ceiling set by the serial fraction. ## Chart Design Measuring how much of a number reaches the reader as it passes through an encoding channel: which channel the comparison, distribution, composition, and relationship questions require, the shape that a histogram bin and a box summary cannot give back from a distribution, the read-back error that stacked and grouped displays produce on slices with no baseline, what a scatter and a heatmap carry from a relationship, separating trend from seasonality on the time axis, the area channel magnifying the ratio by its square, distortions beyond axis truncation, and the number of series that remain indistinguishable when color stands alone. ### Chart Type Selection Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/chart-type-selection The same five regional means are passed through five encoding channels and read back, and the mean read-back error comes out as follows: position 0.070 m³, length 0.183 m³, angle 0.543 m³, color step 0.931 m³, area 2.468 m³. The 4.463 m³ difference between east and north is read back as 2.500 m³ from the area channel and 5.426 m³ from the angle channel, and not a single reading changes. The color step separates four of the five regions, with east and central falling on the same step. Each of the four question types asks for something different to be read back, and the channel pairing follows from that demand. Across thirteen lessons the previous course wrote the margin of variation next to every number, and it closed with this question: all these numbers sit on the analyst's screen as output blocks. The person who will make the decision saw neither the margin nor the number. When a number leaves the screen and goes to a decision maker, it rides on a **display**, and what happens there was never asked. This course opens with that question and changes the measure. In the previous five courses the number varied in the analyst's hands; here the number varies **on its way to the reader**. A display is an **encoding channel**: the number is turned into a length, a position, an angle, an area, or a color step, and the reader turns it back. No picture is produced here; each channel is built with two functions, and both run. The number of the lesson is what the same five regional means look like when read back after passing through five channels. - **CD1. A display is an encoding channel.** A channel is defined by two functions: the **encoder** turns the number into the channel, and the **decoder** reads the number back from the channel. Both sit in the code and both run. - **CD2. Reading resolution is a number.** The reader does not read a channel with infinite precision; each channel has a **number of steps** the reader can tell apart. These numbers are a model assumption written under the output; what is measured is not the number of steps but the difference that comes out of it. - **CD3. The measure is the difference.** The difference between `decoder(encoder(x))` and `x` is the **read-back error**. The number in the source never changes; what varies is the number that reaches the reader. - **CD4. The decoder of the area channel does not read area.** The reader reads the linear measure that carries the shape and reports the value in proportion to it. This is a deviation independent of resolution, produced by the channel itself. - **CD5. Color looks continuous but is stepped.** The value falls into one of a limited number of steps of a legend scale and is read back as the midpoint of that step. Here color is not a name but a **step number**. - **CD6. A channel's success depends on the question.** The same channel can have the absolute value read back badly and the share well. A channel cannot be chosen without writing which question is asked. - **CD7. The decision threshold is written.** A read-back error is large or small only when it is compared with a threshold, and the threshold comes from the decision the report will drive. #### Channel, Encoder and Decoder ```python # channel.py — MODEL: the measurement network of K03-K05 (a fictional setup) is # regenerated with the same seed (a 3,199-row long-format readings table, 1,329 # subscribers, three periods, five regions). No chart is drawn: each display # consists of one encoder and one decoder. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() NAMES = [b[0] for b in REGION] MEAN, OBS = {}, {} for name in NAMES: v = [o["consumption_m3"] for o in READINGS if o["region"] == name] MEAN[name], OBS[name] = sum(v) / len(v), len(v) TOTAL = sum(MEAN.values()) LOW, HIGH = 0.0, 25.0 # Reading resolution: the number of steps the reader can tell apart in that # channel. The numbers are an assumption of the model; what is measured is the # difference that comes out of this assumption. RES = {"position": 100, "length": 40, "angle": 36, "area": 20, "color": 7} def snap(t, n): return round(t * n) / n def encode_position(v): return (v - LOW) / (HIGH - LOW) def decode_position(p): return LOW + (HIGH - LOW) * snap(p, RES["position"]) def encode_length(v): return (v - LOW) / (HIGH - LOW) def decode_length(u): return LOW + (HIGH - LOW) * snap(u, RES["length"]) def encode_angle(v): return 360.0 * v / TOTAL def decode_angle(a): return TOTAL * snap(a / 360.0, RES["angle"]) def encode_area(v): return (v - LOW) / (HIGH - LOW) def decode_area(ar): # The reader reads not the area but the linear measure that carries the # shape: the radius. return LOW + (HIGH - LOW) * snap(math.sqrt(ar), RES["area"]) def encode_color(v): return (v - LOW) / (HIGH - LOW) def decode_color(t): n = RES["color"] step = min(n - 1, int(t * n)) return LOW + (HIGH - LOW) * (step + 0.5) / n CHANNELS = [("position", encode_position, decode_position), ("length", encode_length, decode_length), ("angle", encode_angle, decode_angle), ("area", encode_area, decode_area), ("color", encode_color, decode_color)] print(f"fictional dataset: {len(READINGS)} readings, {len(VALID)} subscribers, " f"{len(NAMES)} regions") print("region means: " + ", ".join(f"{n} {MEAN[n]:.3f}" for n in NAMES) + " m3") print(f"total {TOTAL:.3f} m3, value axis {LOW:.0f}-{HIGH:.0f} m3") print("reading resolution (steps): " + ", ".join(f"{k} {v}" for k, v in RES.items())) ``` ``` fictional dataset: 3199 readings, 1329 subscribers, 5 regions region means: north 20.210, south 16.602, east 24.674, west 13.778, central 22.396 m3 total 97.659 m3, value axis 0-25 m3 reading resolution (steps): position 100, length 40, angle 36, area 20, color 7 ``` The dataset is fictional and is the same regional measurement network the previous three courses used: the same seed, the same 3,199 rows, the same 1,329 subscribers. The region means are the same too; east's 24.674 m³ and north's 20.210 m³ were computed earlier and are taken here as input. The five resolutions were written for these reasons. **Position** is read on a shared axis and a scale stands next to the reader; 100 steps along the axis were taken as distinguishable. **Length** is measured from its own baseline, and the comparison is made against the neighboring bar, not the scale; 40 steps were written. **Angle** is read against a right angle and its subdivisions; a precision of one ninth of it was assumed and the circle was divided into 36 steps. For **area** there are 20 steps, but the real problem is elsewhere: the decoder reads not the area but the linear measure that carries it. **Color** is read from a legend scale, and 7 steps were taken as distinguishable on the scale. These five numbers are debatable and vary from reader to reader. The claim of the lesson is not the numbers themselves but that the **ordering** comes out of them: when the same value passes through these five channels, the numbers read back separate, and the direction of the separation stays the same whatever the resolution choice. #### The Same Number Passes Through Five Channels ```python # readback.py — the same five numbers pass through five channels and are read # back. The measure is the difference between decoder(encoder(x)) and x. G = 40 def bar(v): return "#" * round(G * (v - LOW) / (HIGH - LOW)) print("length channel, 40 steps from the baseline (what the reader sees):") for name in NAMES: print(f"{name:8s}{MEAN[name]:7.3f} |{bar(MEAN[name]):<40s}| " f"read back {decode_length(encode_length(MEAN[name])):6.3f}") print(f"\n{'region':8s}{'count':>7s}{'source':>9s}" + "".join(f"{ch[0]:>10s}" for ch in CHANNELS)) err = {ch[0]: [] for ch in CHANNELS} for name in NAMES: x = MEAN[name] line = f"{name:8s}{OBS[name]:7d}{x:9.3f}" for cname, enc, dec in CHANNELS: g = dec(enc(x)) err[cname].append(abs(g - x)) line += f"{g:10.3f}" print(line) print(f"{'|error|':8s}{sum(OBS.values()):7d}{'':9s}" + "".join(f"{sum(err[ch[0]]) / len(NAMES):10.3f}" for ch in CHANNELS)) ``` ``` length channel, 40 steps from the baseline (what the reader sees): north 20.210 |################################ | read back 20.000 south 16.602 |########################### | read back 16.875 east 24.674 |####################################### | read back 24.375 west 13.778 |###################### | read back 13.750 central 22.396 |#################################### | read back 22.500 region count source position length angle area color north 921 20.210 20.250 20.000 18.989 22.500 19.643 south 645 16.602 16.500 16.875 16.277 20.000 16.071 east 557 24.674 24.750 24.375 24.415 25.000 23.214 west 476 13.778 13.750 13.750 13.564 18.750 12.500 central 600 22.396 22.500 22.500 21.702 23.750 23.214 |error| 3199 0.070 0.183 0.543 2.468 0.931 ``` The bars are the lesson's whole display apparatus: the reader sees an end point on a line of forty steps and derives a number from that end. North's 20.210 m³ fills thirty-two of the forty steps, and thirty-two steps read back as 20.000 m³; the 0.210 m³ in between is lost because there is no place between the thirty-second and thirty-third steps. The mean read-back error of the five channels ranges from 0.070 to 2.468 m³; the best and worst channels differ by a factor of thirty-five. The order is this: position, length, angle, color step, area. The difference between the first two comes from resolution alone. Everything after angle says something else. **The area channel** deserves its own look, because raising the resolution cannot reduce its error. West's 13.778 m³ is read back as 18.750, that is, it rises by 4.972 m³; east's 24.674 m³ stays almost where it is. The deviation is in the same direction for every value, and it looks large because the values are small. The cause is CD4: area grows in proportion to the value, whereas the reader reads not the area but the linear measure that carries it. This is not a misuse of the channel; it is the channel itself. **The color step** gets stuck in a different way. East is read as 23.214 and central is also read as 23.214 — the two regions fell on the same step, and for the reader they are one number, not two. On a seven-step scale the 25 m³ axis is divided into slices of 3.571 m³, and every difference narrower than this slice disappears. The previous course's headline number is tested here. The difference between east and north was 4.463 m³ and had been reported together with its margin of variation. The same difference is read back as **2.500 m³** from the area channel and as **5.426 m³** from the angle channel. Not a single reading in the source has changed; only the channel the number rode did. The `summary-and-cleanup/04` lesson of the Analysis with Spreadsheets course had measured a variation of the same kind with axis truncation, and what was measured there was **the legitimacy of the display**: whether this chart is acceptable at this scale. The measure here is different and works even when the display is entirely legitimate — a pie chart, a bubble chart, and a heatmap are all within the rules, and they still move the number. #### The Question Type Selects the Channel ```python # question.py — four question types, four separate read-backs. A channel's # fitness for a question is whether what the question asks for is read back # within the decision threshold. THRESHOLD_DIFF, THRESHOLD_SHARE = 0.5, 1.0 diff = MEAN["east"] - MEAN["north"] share = {name: MEAN[name] / TOTAL for name in NAMES} print(f"source difference east-north {diff:.3f} m3; decision threshold " f"{THRESHOLD_DIFF} m3, share threshold {THRESHOLD_SHARE} points") print(f"{'channel':9s}{'read-back difference':>22s}{'error':>8s}{'distinct':>10s}" f"{'largest share error':>21s}") result = {} for cname, enc, dec in CHANNELS: g = {name: dec(enc(MEAN[name])) for name in NAMES} gd = g["east"] - g["north"] t = sum(g.values()) gs = max(abs(100 * g[name] / t - 100 * share[name]) for name in NAMES) distinct = len({round(g[name], 6) for name in NAMES}) result[cname] = (abs(gd - diff) <= THRESHOLD_DIFF, distinct == len(NAMES), gs <= THRESHOLD_SHARE) print(f"{cname:9s}{gd:22.3f}{abs(gd - diff):8.3f}{distinct:8d}/5{gs:21.2f}") QUESTIONS = [("comparison", "difference of two values"), ("distribution", "order of five values"), ("composition", "share of each value")] print(f"\n{'question':16s}{'must be read back':24s}" + "".join(f"{ch[0]:>9s}" for ch in CHANNELS)) for j, (name, what) in enumerate(QUESTIONS): print(f"{name:16s}{what:24s}" + "".join(f"{'yes' if result[ch[0]][j] else 'no':>9s}" for ch in CHANNELS)) print(f"{'relationship':16s}{'two values at once':24s}" + "".join(f"{'yes' if ch[0] == 'position' else 'no':>9s}" for ch in CHANNELS)) ``` ``` source difference east-north 4.463 m3; decision threshold 0.5 m3, share threshold 1.0 points channel read-back difference error distinct largest share error position 4.500 0.037 5/5 0.12 length 4.375 0.088 5/5 0.31 angle 5.426 0.962 5/5 0.69 area 2.500 1.963 5/5 2.94 color 3.571 0.892 4/5 1.60 question must be read back position length angle area color comparison difference of two values yes yes no no no distribution order of five values yes yes yes yes no composition share of each value yes yes yes no no relationship two values at once yes no no no no ``` Two thresholds were written and both come from the report. **0.5 m³** comes from the fact that the region difference is written into the report with two decimals and that the decision on which region the field team will visit is made on a difference of one m³. **1.0 point** comes from the share being written in the report as a whole-number percentage: a one-point error changes the written digit. Without a written threshold, no read-back error is large or small. The first row of the table is the expected one: the comparison question is carried only by position and length. The second row passes most of the channels, because preserving the order is the lightest job that can be asked of a channel; only the color step fails, for exactly the reason seen above: two regions fell on the same step. The third row is unexpected. **The angle channel passes the share but not the difference.** The cause is in the encoder: angle already encodes the share, not the absolute value. The resolution of angle is spent on the share, and while the share error stays at 0.69 points, the absolute difference error climbs to 0.962 m³. The reverse also holds: of the four questions, the area channel passes only the order, because the CD4 deviation pushes every value in the same direction and the order is preserved, but it breaks both the difference and the share, and breaks the share most, by 2.94 points. The fourth row is not measured, it is counted. The relationship question asks for two values to be read back **at the same time**, and for this a second independent copy of the channel is needed. Only position has two copies, the horizontal and the vertical axis; length, angle, area, and color have no second copy. So the relationship question is not a channel preference but a channel requirement. The rule that comes out of this is the concrete form of CD6: the channel is chosen from the question, not from the data. The same five numbers answer "which is larger" from four channels, "how much larger" from two, and "what part of the total" from three. #### Summary - A display is modeled with two functions: the encoder turns the number into the channel, the decoder reads it back; the measure is the read-back error between their composition and the source number. - Reading resolution is written as a number — position 100, length 40, angle 36, area 20, color 7 steps — and these are a model assumption; what is measured is the difference they produce. - The mean read-back error of the same five regional means comes out as 0.070 for position, 0.183 for length, 0.543 for angle, 0.931 for the color step, and 2.468 m³ for area. - The error of area cannot be reduced by resolution, because the decoder reads not the area but the linear measure; the color step separates only four of the five regions, and east and central fall on the same step. - The 4.463 m³ difference between east and north is read back as 2.500 m³ from area and 5.426 m³ from angle, and not a single reading in the source changes. - The channel is chosen from the question, not from the data: angle passes the share but not the difference, while the relationship question asks for a second independent copy of the channel, and that copy exists only in position. #### Next Step In this lesson every region was reduced to a single number, and the channel carried it. Some of the questions in the report, however, are not answered by a single number: how the 921 readings of north are laid out, whether there are two peaks or one, where the extreme values stand. The next lesson applies the same measure to a **distribution** and compares three displays — bin counts, the box summary, and the density display. What is read back is no longer a number but a **distributional feature**, and the measure is which display reads back the second peak. That the bin width is a decision, and that this decision moves the number of peaks read back, is shown with numbers. ### Distribution Visualization Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/distribution-visualization The twin set that gives the same five numbers as the 921 readings of north is passed through three displays, and what is read back is not a number but a distributional feature: bin counts and the density display read back the two peaks of the twin, while the box summary carries no peak information at all. In the counts that pass through the length channel three bins drop to zero and 7 readings are erased from the display, and the 45.47 m³ extreme value of the twin disappears; the same extreme value is read back in north, because the tallest bar is 234, not 329. When the bin width is 12 m³ the number of peaks read back falls from two to one, and when the kernel width is 0.5 m³ four peaks that are not in north appear. The previous lesson reduced every region to a single number, and the channel carried that single number. Some of the questions in the report cannot be answered by a single number: are the 921 readings of north gathered around a single value or are there two separate groups of subscribers, and do the largest readings stand in the display. These questions look at a **distribution**, and a distribution does not fit in a single position. In the previous course this distribution had been built as numbers: the five-number summary was computed and a second set giving the same five numbers was produced. That computation is not repeated; it is taken as input. The question of this lesson is a step further: when the same distribution is encoded with three separate displays, **which distributional feature** can the reader read back. - **CD8. A distribution display is also a channel.** Bin counts put the counts on the **length** channel; the box summary and the density display put the values on the **position** channel. The resolutions of the previous lesson continue as they are: length 40, position 100 steps. - **CD9. What is read back is a distributional feature.** Three features are counted: the **number of peaks**, the **skew direction**, and **whether the extreme value is visible**. - **CD10. The peak rule is written.** A point is a peak if it is larger than both of the first different values on its two sides and is not below 0.3 times the highest value. The rule is the same in all three displays and is applied to the **read-back** values. - **CD11. The bin width and the kernel width are decisions.** Both change not the source but the shape that is read back. - **CD12. The visibility of the extreme value does not depend on the extreme value.** The bars are scaled to the highest count; the same extreme value disappears in a display whose neighbor is higher. - **CD13. The box summary carrying no peak is not a flaw but its scope.** Five numbers report five points of the ordering; how the values are laid out between those points never enters the box. #### The Shape in the Source ```python # display.py — MODEL: the measurement network of K03-K05 (a fictional setup) is # generated with the same seed. The 921 readings of north and the twin set of # K05 are the input; neither is recomputed here, both are passed through the # display channel. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() north = sorted(o["consumption_m3"] for o in READINGS if o["region"] == "north") def push(chunk, end, p=4): a, z = chunk[0], chunk[-1] u = [(x - a) / (z - a) for x in chunk] return [z - (z - a) * (1 - t) ** p if end == "high" else a + (z - a) * t ** p for t in u] CUTS = [0, 230, 460, 690, len(north) - 1] twin = (push(north[CUTS[0]:CUTS[1] + 1], "high")[:-1] + push(north[CUTS[1]:CUTS[2] + 1], "low")[:-1] + push(north[CUTS[2]:CUTS[3] + 1], "high")[:-1] + push(north[CUTS[3]:], "low")) # Channels: the bin-count bar is length; the box and the density are position # on a shared axis. RES = {"length": 40, "position": 100} def snap(t, n): return round(t * n) / n def decode_length(v, top): return snap(v / top, RES["length"]) * top if top > 0 else 0.0 def decode_position(v, low, top): return low + (top - low) * snap((v - low) / (top - low), RES["position"]) def percentile(v, p): s = sorted(v) i = p * (len(s) - 1) a = int(i) b = min(a + 1, len(s) - 1) return s[a] + (s[b] - s[a]) * (i - a) # Shape in the source: the number of peaks is known from the setup itself (north # was generated from a single base, the twin was pushed to two ends). The skew # direction is the sign of mean minus median; the threshold is 0.05 m3, because # a difference smaller than this does not reach the text in a number written # with two decimals. Extreme value: the zone in which the largest observation # falls. THRESHOLD_SKEW = 0.05 SETS = [("north", north, 1), ("twin", twin, 2)] def direction(d): return "right" if d > THRESHOLD_SKEW else ("left" if d < -THRESHOLD_SKEW else "symmetric") SOURCE = {} for name, v, peaks in SETS: mean, med = sum(v) / len(v), percentile(v, 0.50) SOURCE[name] = (peaks, direction(mean - med), max(v)) print(f"{name:6s} n={len(v)} mean {mean:.3f} median {med:.3f} " f"largest {max(v):.2f} m3") print(f" shape in the source: peaks {peaks}, skew {direction(mean - med)}, " f"extreme value {max(v):.2f} m3") ``` ``` north n=921 mean 20.210 median 19.660 largest 45.47 m3 shape in the source: peaks 1, skew right, extreme value 45.47 m3 twin n=921 mean 19.754 median 19.660 largest 45.47 m3 shape in the source: peaks 2, skew right, extreme value 45.47 m3 ``` The number of peaks in the source is not estimated, it is **known**: both sets are fictional, and how they were built is in the code. The 921 readings of north were generated from a single base value; the twin set was obtained by pushing the sorted values of north to two ends in quarter slices — one has one gathering center, the other has two. The skew direction is read from the sign of mean minus median; the threshold was written as 0.05 m³, because a smaller difference does not make it into the text in a number reported with two decimals. In north the difference is 0.550 m³, in the twin 0.094 m³, and the second is just above the threshold. #### The Same Distribution Passes Through Three Displays ```python # three_displays.py — the same distribution is encoded with three displays and # three distributional features are tried to be read back. The peak rule is the # same in every display: a point larger than the different values on its two # sides and not below 0.3 times the highest. WIDTH, AXIS, N = 4.0, 48.0, 12 H, M = 2.0, 60 def bin_counts(v, width, n): c = [0] * n for x in v: c[min(int(x / width), n - 1)] += 1 return c def peaks(y, ratio=0.3): m, t = max(y), [] for i in range(len(y)): if i and y[i] == y[i - 1]: continue before = next((y[j] for j in range(i - 1, -1, -1) if y[j] != y[i]), None) after = next((y[j] for j in range(i + 1, len(y)) if y[j] != y[i]), None) if before is not None and after is not None and y[i] > before and y[i] > after and y[i] >= ratio * m: t.append(i) return t def summarize(x, w): tot = sum(w) mean, cum = sum(a * b for a, b in zip(x, w)) / tot, 0.0 for a, b in zip(x, w): cum += b if cum >= tot / 2: return mean, a return mean, x[-1] def display_bins(v, width=WIDTH): n = round(AXIS / width) c = bin_counts(v, width, n) g = [round(decode_length(x, max(c))) for x in c] mean, med = summarize([(i + 0.5) * width for i in range(n)], g) return len(peaks(g)), direction(mean - med), g[min(int(max(v) / width), n - 1)] > 0, g def display_box(v): b = [min(v), percentile(v, 0.25), percentile(v, 0.50), percentile(v, 0.75), max(v)] g = [decode_position(x, 0.0, AXIS) for x in b] iqr = g[3] - g[1] low_fence, high_fence = g[1] - 1.5 * iqr, g[3] + 1.5 * iqr outside = [x for x in v if x < low_fence or x > high_fence] dots = len({decode_position(x, 0.0, AXIS) for x in outside}) return None, direction((g[3] - g[2]) - (g[2] - g[1])), dots > 0, (len(outside), dots) def display_density(v, h=H): x = [AXIS * i / (M - 1) for i in range(M)] y = [sum(math.exp(-0.5 * ((a - t) / h) ** 2) for t in v) for a in x] g = [decode_position(t, 0.0, max(y)) for t in y] mean, med = summarize(x, g) return (len(peaks(g)), direction(mean - med), g[round((M - 1) * max(v) / AXIS)] > 0, g) DISPLAYS = [("bin counts", display_bins), ("box summary", display_box), ("density", display_density)] print(f"bin width {WIDTH:.0f} m3, kernel width {H:.0f} m3, " f"axis 0-{AXIS:.0f} m3, {M} sampling points") _c = bin_counts(twin, WIDTH, N) _g = [round(decode_length(x, max(_c))) for x in _c] print("\ntwin set, when the counts pass through the length channel (source / read back):") for i in range(N): print(f"{i * WIDTH:5.0f}-{(i + 1) * WIDTH:<3.0f}{_c[i]:5d}{_g[i]:6d} " f"|{'#' * round(40 * _g[i] / max(_g))}") _lost = [(i, _c[i]) for i in range(N) if _c[i] > 0 and _g[i] == 0] print(f"bins dropped to zero on read-back: {len(_lost)}, " f"readings inside them: {sum(x for _, x in _lost)}\n") print(f"{'set':7s}{'display':16s}{'peaks':>12s}{'skew':>12s}{'extreme value':>14s}" f"{'features passed':>17s}") for name, v, _ in SETS: kp, kd, _ke = SOURCE[name] for d_name, f in DISPLAYS: t, c, u, _ex = f(v) passed = sum([t == kp, c == kd, u]) print(f"{name:7s}{d_name:16s}{str(t) if t is not None else 'unreadable':>12s}" f"{c:>12s}{'yes' if u else 'no':>14s}{passed:>14d}/3") ``` ``` bin width 4 m3, kernel width 2 m3, axis 0-48 m3, 60 sampling points twin set, when the counts pass through the length channel (source / read back): 0-4 32 33 |#### 4-8 0 0 | 8-12 0 0 | 12-16 318 321 |####################################### 16-20 114 115 |############## 20-24 110 107 |############# 24-28 329 329 |######################################## 28-32 11 8 |# 32-36 4 0 | 36-40 0 0 | 40-44 2 0 | 44-48 1 0 | bins dropped to zero on read-back: 3, readings inside them: 7 set display peaks skew extreme value features passed north bin counts 1 right yes 3/3 north box summary unreadable right yes 2/3 north density 1 right yes 3/3 twin bin counts 2 right no 2/3 twin box summary unreadable right yes 2/3 twin density 2 left no 1/3 ``` The bar table shows directly what the channel does. A bar of forty steps is scaled to the highest count; in the twin the highest count is 329, so one step corresponds to 8.2 readings. Every bin with fewer than five readings stays below half a step and is read back as zero. Three bins are erased this way, and the **7 readings** inside them leave the display entirely; the 45.47 m³ extreme value is among them. The same extreme value is read back in north, and the only reason is CD12: in north the highest count is 234, one step is 5.9 readings, and the three readings in the 44–48 m³ bin are enough to fill a step. The extreme value is in the same place and at the same count in both sets; what determines whether it appears is **the height of the tallest bar**. In the feature table no display passes all three, and each one captures something the others miss. The **box summary** carries no peak count at all — the column says "unreadable", because there is no way to derive a peak from five numbers. In return it is the only display that has the extreme value read back in both sets: the observations outside the fence are printed as separate dots, they are not scaled, and they do not look at their neighbors. **Bin counts and density** are the only displays that read back the two peaks of the twin, and both lose the extreme value, because both are scaled to the highest value. That the density display reads the skew of the twin as **reversed** is not a display flaw but the result of a source that sits close to the threshold: an asymmetry of 0.094 m³ cannot keep its direction when it is reduced to sixty sampling points and a position grid of a hundred steps. For a distributional feature this close to the decision threshold, the display choice determines the result, and that feature is not reported. #### The Bin Width Is a Decision ```python # decision.py — the bin width and the kernel width are decisions; both move the # number of peaks read back. The number of peaks in the source does not change. print(f"{'width':>9s}{'bins':>8s}" + "".join(f"{name:>9s}" for name, _, _ in SETS)) for width in (2.0, 3.0, 4.0, 6.0, 8.0, 12.0): print(f"{width:8.0f} m3{round(AXIS / width):8d}" + "".join(f"{display_bins(v, width)[0]:9d}" for _, v, _ in SETS)) print(f"{'kernel':>9s}{'samples':>8s}" + "".join(f"{name:>9s}" for name, _, _ in SETS)) for h in (0.5, 1.0, 2.0, 3.0, 5.0, 8.0): print(f"{h:8.1f} m3{M:8d}" + "".join(f"{display_density(v, h)[0]:9d}" for _, v, _ in SETS)) print("peaks in the source: " + ", ".join(f"{name} {t}" for name, _, t in SETS)) ``` ``` width bins north twin 2 m3 24 1 2 3 m3 16 1 2 4 m3 12 1 2 6 m3 8 1 2 8 m3 6 1 2 12 m3 4 1 1 kernel samples north twin 0.5 m3 60 4 2 1.0 m3 60 1 2 2.0 m3 60 1 2 3.0 m3 60 1 2 5.0 m3 60 1 1 8.0 m3 60 1 1 peaks in the source: north 1, twin 2 ``` The number of peaks in the source does not change in either row: one in north, two in the twin. The only thing that changes is the number read back, and the only thing that changes it is a decision. With the bin width between 2 and 8 m³, the two peaks of the twin are read back. At 12 m³ the number falls to one, because the distance between the two gathering centers is about 12 m³ and a single bin swallows both. The previous course had pointed at this trap with a single sentence; here the place of the threshold is counted. The kernel width goes wrong in the opposite direction as well. At 0.5 m³, **four** peaks are read back in north, although there is one in the source. At this width the kernel is narrower than the 0.81 m³ gap between sampling points; instead of smoothing the data the curve follows individual readings, and random fluctuations in the reading density are read back as peaks. At 5 m³ and above, the two peaks of the twin merge. Between the two decisions there is not a preference but two separate errors: a narrow choice produces the peak that is not there, a wide choice erases the one that is. The rule is the concrete form of CD11 — both widths are written under the output, otherwise the number of peaks read back looks like a finding. #### Summary - Bin counts put the counts on the length channel, while the box summary and the density display put the values on the position channel; what is read back is not an m³ value but the number of peaks, the skew direction, and the extreme value. - No display passes all three features: the box summary carries no peak at all but is the only display that has the extreme value read back in both sets; bin counts and density read back the two peaks but lose the extreme value. - In the counts that pass through the length channel three bins drop to zero and 7 readings are erased; the same 45.47 m³ extreme value shows in north because the tallest bar there is 234, not 329. - A bin width of 12 m³ lowers the number of peaks read back in the twin from two to one, and a kernel width of 0.5 m³ shows four peaks in north that are not there; the number of peaks in the source does not change. - The skew of 0.094 m³ in the twin is just above the threshold and its direction changes with the display; a distributional feature this close to the threshold is not reported. #### Next Step In this lesson there was a continuous value on one side of the axis, and the display turned it into bins or a curve. Most of the questions in the report, however, are **categorical**: the number of subscribers in the four tariff tiers of five regions and the share of those numbers within the region. Categorical display has a decision of its own — whether the slices will be stacked or placed side by side — and this is a channel problem: in a stacked display only the bottom slice has a baseline. The next lesson counts the read-back error of the upper slices by slice order and compares it with the grouped display. ### Categorical Visualization Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/categorical-visualization The crosstab of five regions and four tariff tiers is encoded with two displays, and the difference shows up at the baseline: in the stacked display the bottom slice is read from the position channel and its relative error is 2.7%, while the upper slices are read from the length channel and the error is 22.0% in the third slice and 47.2% in the fourth. North's fourth slice of 8 readings and west's third slice of 11 readings are read back as zero, that is, they leave the display entirely. Of the 39 region pairs that must be separated, the stacked display separates 35 and the grouped display 39; no pair is read reversed, so what is lost is not the order but the distinction. The same stacked display gives the total with a single reading: 921 observations come back as 920. The previous lesson encoded the distribution of a continuous value with three displays. Some of the questions in the report are not continuous but **categorical**: the number of readings of five regions in four tariff tiers, the share of those numbers within the region, the order of a tier among the regions. Categorical data sits in a crosstab, and that table can be displayed in two different ways — the slices are either stacked on top of each other or placed side by side. The difference between these two forms is not a matter of style but a difference of channel. In the stacked display only the bottom slice starts from the zero of the axis; the others float above the axis, and the reader has no starting point to measure them from. The numbers of the lesson are the counterpart of this difference in observations. - **CD14. The stacked display has a single baseline.** The bottom slice starts from the zero of the axis and is read from the **position** channel; the lower edge of the upper slices is not on the axis but at the point where the previous slice ends. - **CD15. The floating slice is read from the length channel.** The reader does not read two positions and subtract; the reader looks at the length of the slice. The resolutions of the first lesson continue: position 100, length 40 steps. - **CD16. The axis span comes from the display.** The stacked axis has to carry the **total**, the grouped axis only the **largest slice**. The same number of steps means a step of a different size in the two displays. - **CD17. A slice smaller than half an axis step is read back as zero.** A slice that sits in the source leaves the display entirely and reaches the reader as "none". - **CD18. The comparison of a categorical question is between categories.** The measure is **how many of the region pairs that must be separated are separated**, in the same tier. - **CD19. The stacked display gives the total with a single reading.** The upper edge of the stack is measured from the zero of the axis; the number that edge carries is the total, not the parts. #### The Crosstab and the Two Axes ```python # categorical.py — MODEL: the measurement network of K03-K05 is # generated with the same seed. The crosstab of five regions and four tariff # tiers is the input; this table will be encoded with two separate displays # and read back. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() TARIFF = [10, 25, 40] NAME = [b[0] for b in REGION] def tier(x): return (1 if x < TARIFF[0] else 2 if x < TARIFF[1] else 3 if x < TARIFF[2] else 4) K = len(TARIFF) + 1 CROSS = {a: [0] * K for a in NAME} for o in READINGS: CROSS[o["region"]][tier(o["consumption_m3"]) - 1] += 1 TOTAL = {a: sum(CROSS[a]) for a in NAME} # The axis span changes with the display: the stacked axis has to carry the # total, the grouped axis only the largest slice. The upper limit is rounded up # to the next hundred, because the axis label sits on round numbers. RES_POSITION = 100 def round_hundred(x): return (int(x) // 100 + 1) * 100 AXIS_STACKED = round_hundred(max(TOTAL.values())) AXIS_GROUPED = round_hundred(max(max(CROSS[a]) for a in NAME)) def grid(v, axis, n): return axis * round(n * v / axis) / n print(f"fictional crosstab: {len(READINGS)} readings, tariff tiers " f"{TARIFF[0]}/{TARIFF[1]}/{TARIFF[2]} m3") print(f"{'region':8s}" + "".join(f"{'tier ' + str(i + 1):>12s}" for i in range(K)) + f"{'total':>9s}") for a in NAME: print(f"{a:8s}" + "".join(f"{c:12d}" for c in CROSS[a]) + f"{TOTAL[a]:9d}") print(f"stacked axis 0-{AXIS_STACKED}, one step {AXIS_STACKED / RES_POSITION:.1f} observations; " f"grouped axis 0-{AXIS_GROUPED}, one step {AXIS_GROUPED / RES_POSITION:.1f} observations") ``` ``` fictional crosstab: 3199 readings, tariff tiers 10/25/40 m3 region tier 1 tier 2 tier 3 tier 4 total north 42 661 210 8 921 south 59 538 48 0 645 east 21 281 225 30 557 west 88 377 11 0 476 central 30 352 198 20 600 stacked axis 0-1000, one step 10.0 observations; grouped axis 0-700, one step 7.0 observations ``` The crosstab is the source itself and carries twenty numbers. The magnitudes of the numbers are very different: the second tier varies between 281 and 661, the fourth tier between 0 and 30. The difficulty of categorical display starts here, because a single axis has to carry both magnitudes at the same time. The axis spans of the two displays are different, and the difference is not a preference but a necessity. In the stacked display the bar of a region extends as far as the total of all slices, so the axis has to carry the largest total, 921, and ends at the next round number, 1000. In the grouped display the bars stand side by side and the axis carries only the largest slice, 661; the upper limit is 700. The same position grid of a hundred steps means a step of 10 observations in one and a step of 7 observations in the other. #### Stacked: The Slice Without a Baseline ```python # stacked.py — the slices are placed on top of each other. The bottom slice # starts from the zero of the axis and is read from the position channel; the # upper slices float above the axis, and instead of subtracting two positions # the reader looks at the length of the slice and reads it with the 40 steps # of the length channel (first lesson). STEPS, RES_LENGTH = 40, 40 def read_stacked(c): return [grid(c[0], AXIS_STACKED, RES_POSITION)] + [ grid(x, AXIS_STACKED, RES_LENGTH) for x in c[1:]] def read_grouped(c): return [grid(x, AXIS_GROUPED, RES_POSITION) for x in c] def bar(c): s, t = "", 0.0 for i, x in enumerate(c): t += x s += str(i + 1) * (round(STEPS * t / AXIS_STACKED) - len(s)) return s print(f"stacked display, {STEPS}-step axis (what the reader sees):") for a in NAME: print(f"{a:8s}|{bar(CROSS[a]):<40s}| total {TOTAL[a]}") print(f"\n{'region':8s}" + "".join(f"{'slice ' + str(i + 1):>14s}" for i in range(K))) abs_err = [[] for _ in range(K)] rel_err = [[] for _ in range(K)] lost = 0 for a in NAME: g = read_stacked(CROSS[a]) for i in range(K): abs_err[i].append(abs(g[i] - CROSS[a][i])) if CROSS[a][i] > 0: rel_err[i].append(100 * abs(g[i] - CROSS[a][i]) / CROSS[a][i]) lost += g[i] == 0 print(f"{a:8s}" + "".join(f"{c:>7d}/{x:<6.0f}" for c, x in zip(CROSS[a], g))) print(f"\n{'slice':6s}{'channel':>10s}{'|error|':>9s}{'relative':>10s}") for i in range(K): print(f"{i + 1:<6d}{'position' if i == 0 else 'length':>10s}" f"{sum(abs_err[i]) / len(NAME):9.1f}{sum(rel_err[i]) / len(rel_err[i]):9.1f}%") print(f"slices read back as zero though above zero in the source: {lost}") _t = grid(max(TOTAL.values()), AXIS_STACKED, RES_POSITION) print(f"the total comes with a single reading in the same display: north {TOTAL['north']} " f"-> {_t:.0f}") ``` ``` stacked display, 40-step axis (what the reader sees): north |1122222222222222222222222222333333333 | total 921 south |11222222222222222222222233 | total 645 east |1222222222223333333334 | total 557 west |1111222222222222222 | total 476 central |122222222222222333333334 | total 600 region slice 1 slice 2 slice 3 slice 4 north 42/40 661/650 210/200 8/0 south 59/60 538/550 48/50 0/0 east 21/20 281/275 225/225 30/25 west 88/90 377/375 11/0 0/0 central 30/30 352/350 198/200 20/25 slice channel |error| relative 1 position 1.2 2.7% 2 length 6.6 1.4% 3 length 5.0 22.0% 4 length 3.6 47.2% slices read back as zero though above zero in the source: 2 the total comes with a single reading in the same display: north 921 -> 920 ``` The bars themselves are the clearest evidence of the lesson. North's fourth slice, that is, the eight subscribers with readings above 40 m³, does not show up at all in the bar of forty steps; west's third slice does not either. These are not numbers the display erases but numbers the display cannot carry: on an axis of 1000 units, in a bar divided into 40 steps, one step is worth 25 observations, and 11 observations are below half a step. The error table is read by slice order. The bottom slice passes through the position channel and its relative error stays at 2.7%. The upper slices pass through the length channel, and the relative error rises to 22.0% in the third slice and 47.2% in the fourth. The 1.4% error of the second slice seems to break this pattern but does not: the second slice is the largest slice of the set and has the highest absolute error, 6.6 observations — it is because it is large that its relative error stays small. **Two things determine the error: whether the slice has a baseline, and the size of the slice relative to the axis.** The one thing the same display does not lose is the **total**. The upper edge of the stack is measured from the zero of the axis, comes with a single position reading, and north's 921 observations are read back as 920. This is why the stacked display exists: it gives the total and the size order of the totals at a single glance. It manages to give the parts only for the bottom slice. An objection stands here: a careful reader can read both edges of a slice on the axis and subtract, and get a better number. True, but that reader is no longer looking at the chart; the reader is doing arithmetic on top of the chart. If a number has to be obtained by subtraction, a table takes the place of the display. #### Grouped: A Baseline for Every Slice ```python # grouped.py — the same numbers are placed side by side: every slice starts # from its own baseline, all are read from the position channel, and the axis # carries only the largest slice. The question is between regions: which # region is higher in the same tier. from itertools import combinations print(f"{'tier':8s}{'source':>26s}{'stacked':>26s}{'grouped':>26s}") SERIES = [] for i in range(K): s = ([CROSS[a][i] for a in NAME], [read_stacked(CROSS[a])[i] for a in NAME], [read_grouped(CROSS[a])[i] for a in NAME]) SERIES.append(s) print(f"{i + 1:<8d}" + "".join(f"{x:>26s}" for x in ["/".join(f"{v:.0f}" for v in t) for t in s])) print(f"\n{'tier':8s}{'pairs to separate':>24s}{'stacked':>10s}" f"{'grouped':>8s}{'reversed':>13s}") tally = [0, 0, 0, 0] for i, (k, y, g) in enumerate(SERIES): c = [(p, q) for p, q in combinations(range(len(NAME)), 2) if k[p] != k[q]] sep_y = sum(1 for p, q in c if y[p] != y[q]) sep_g = sum(1 for p, q in c if g[p] != g[q]) rev = sum(1 for p, q in c if (k[p] - k[q]) * (y[p] - y[q]) < 0) for j, v in enumerate((len(c), sep_y, sep_g, rev)): tally[j] += v print(f"{i + 1:<8d}{len(c):>24d}{sep_y:>10d}{sep_g:>8d}{rev:>13d}") print(f"{'total':8s}{tally[0]:>24d}{tally[1]:>10d}{tally[2]:>8d}{tally[3]:>13d}") ``` ``` tier source stacked grouped 1 42/59/21/88/30 40/60/20/90/30 42/56/21/91/28 2 661/538/281/377/352 650/550/275/375/350 658/539/280/378/350 3 210/48/225/11/198 200/50/225/0/200 210/49/224/14/196 4 8/0/30/0/20 0/0/25/0/25 7/0/28/0/21 tier pairs to separate stacked grouped reversed 1 10 10 10 0 2 10 10 10 0 3 10 9 10 0 4 9 6 9 0 total 39 35 39 0 ``` The comparison question is between categories, and the measure is how many of the region pairs that differ in the source are still read as different after the display. In the fourth tier nine of the ten pairs are different in the source — south and west really are the same, zero. The stacked display separates six of these nine, the grouped display all nine. The three pairs the stacked display loses are clear. East's 30 observations and central's 20 observations are both read as 25, and north's 8 observations drop to zero and become the same as the two regions that are really zero. In the third tier west's 11 observations also drop to zero but collide with no one; the only loss there is that north's 210 and central's 198 are both read as 200, and nine of the ten pairs are separated instead of nine. Out of the 39 pairs that must be separated, the stacked display separates 35 and the grouped display 39. The last column is the subtle finding of this lesson: **no pair is read reversed**. The stacked display never shows a small region as big; the only thing it does is erase differences. This comes from the error having no direction — rounding to the grid brings two regions closer to each other, but does not swap them. The sentence to write into the report comes from here: the ranking read from the stacked display is not wrong, it is **incomplete**. How many steps of the ranking can be read is a separate number, and that number is 35 of 39. The price of the grouped display is not in the read-back but in the space: four slices in five regions mean twenty bars, and when one more slice is added it becomes twenty-five. The stacked display fits the same numbers into five bars. The choice between the two displays depends on whether the question asked is about the total or about the parts, and that question is the question of the report, not of the data. #### Summary - In the stacked display only the bottom slice has a baseline; that slice is read from the position channel, the floating slices from the length channel, and since the axis carries the total, one step rises to 25 observations. - The relative read-back error by slice order is 2.7% · 1.4% · 22.0% · 47.2%; two things determine the error: whether the slice has a baseline, and the size of the slice relative to the axis. - North's fourth slice of 8 readings and west's third slice of 11 readings are read back as zero, that is, two slices that sit in the source leave the display entirely. - Of the 39 region pairs that must be separated, the stacked display separates 35 and the grouped display 39; no pair is read reversed, what is lost is not the order but the distinction. - The stacked display gives the total with a single reading — north's 921 observations are read back as 920 — and what it loses is only the parts. #### Next Step In three lessons the categories were counted and the distributions were divided into intervals, and each time there was a label on one side of the axis and a number on the other. Some of the questions in the report ask for two numbers at once: whether the tier also rises as consumption rises in a region, whether the readings of two periods move together. The first lesson had shown that this question can be answered only with the position channel, because the second independent copy of the channel exists only there. The next lesson uses those two copies and counts what happens as the number of points grows: how much overplotting there is, where the unreadable region starts, and how many separate levels are read back when the same relationship is put on the color steps of a heat map. ### Relationship Visualization Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/relationship-visualization When 1,101 subscriber pairs are encoded as a scatter plot, the mark width leaves 1,600 distinguishable cells, and 792 of the points, that is 71.9%, overlap and can no longer be read back; the densest cell holds 16 points. When the same pairs are put on a heat map, not a single point is lost but the count is: the 28 distinct count values in the source fall to six of the seven color steps, 37 of the 51 filled cells land on a single step, and the mean relative read-back error comes out at 280.9%. When the steps are divided into an equal number of cells instead of equal intervals, the same error falls to 15.7%. A line that gives a single y for each x reduces the cloud's mean span of 44.1 steps to a single step. The first lesson, while binding the four question types to channels, had set the relationship question apart: this question asks for two values to be read back **at the same time**, and for that a second independent copy of the channel is needed. Only position has two copies, the horizontal and the vertical axis. This lesson uses those two copies. The pair used is the natural question of the measurement network: how well a subscriber's **first period reading** reports the subscriber's **general level**. The strength of the relationship was measured with a coefficient in the previous course, and here it is neither computed nor interpreted. The question here belongs to display: when the same relationship is encoded in three forms, what reaches the reader, and what is lost as the number of points grows. - **CD20. The relationship question asks for two position channels at the same time.** It is the result of the first lesson and is used here as input. - **CD21. The drawn point is a mark, and it has a width.** The value of a single point is read from the position channel with 100 steps, but **two points** are distinguished only when they are apart by more than the mark width. The mark width was taken as one fortieth of the axis. - **CD22. An overlapping point is not read back.** The reader sees a cell once; the measure is how many points remain in a separate cell. - **CD23. The heat map does not lose points, it loses counts.** Every point is counted in a cell, but the count descends to a color step, and the number of levels that can be read back is limited by the number of steps. - **CD24. The color step is divided not by the distribution of the counts but by the boundary that is set.** If the counts are skewed, in equal-interval steps the lower steps fill up and the upper steps stay empty. - **CD25. A display that gives a single y for an x does not carry the span at that x.** The number the line has read back is one step; the number of steps the cloud covers is a separate measure. #### Two Position Channels and the Mark Width ```python # relationship.py — MODEL: the measurement network of K03-K05 is # generated with the same seed. The relationship pair: the subscriber's first # period reading and the same subscriber's mean. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() from collections import defaultdict # The first is the subscriber's first period reading, the second is the mean # of all the periods of the same subscriber. Subscribers with at least two # readings are taken. _by_subscriber = defaultdict(list) for o in READINGS: _by_subscriber[o["subscriber_id"]].append((o["period"], o["consumption_m3"])) PAIRS = [] for _id in sorted(_by_subscriber): v = sorted(_by_subscriber[_id]) if len(v) >= 2: PAIRS.append((v[0][1], sum(x for _, x in v) / len(v))) # Two points are distinguished only when they are apart by more than the mark # width. The drawn point is a mark and it has a width; the mark width was taken # as one fortieth of the axis. The value of a single point, on the other hand, # is read from the position channel, with 100 steps. RES_POSITION, MARK = 100, 40 def round_ten(x): return (int(x) // 10 + 1) * 10 AXIS_X = round_ten(max(a for a, _ in PAIRS)) AXIS_Y = round_ten(max(b for _, b in PAIRS)) def cell(p): return (min(MARK - 1, int(MARK * p[0] / AXIS_X)), min(MARK - 1, int(MARK * p[1] / AXIS_Y))) # The choice of the subset comes from the seed: the pair list is shuffled once # and then taken from the start. This way every size is a prefix of the same # order. _r = rng(SEED + 404) ORDER = list(range(len(PAIRS))) for i in range(len(ORDER) - 1, 0, -1): j = int(_r() * (i + 1)) ORDER[i], ORDER[j] = ORDER[j], ORDER[i] SHUFFLED = [PAIRS[i] for i in ORDER] print(f"fictional relationship set: {len(PAIRS)} subscriber pairs " f"(first period reading, subscriber mean)") print(f"axes 0-{AXIS_X} and 0-{AXIS_Y} m3; position resolution {RES_POSITION} " f"steps, mark width 1/{MARK} of the axis") print(f"plane {MARK}x{MARK} = {MARK * MARK} distinguishable cells") ``` ``` fictional relationship set: 1101 subscriber pairs (first period reading, subscriber mean) axes 0-60 and 0-50 m3; position resolution 100 steps, mark width 1/40 of the axis plane 40x40 = 1600 distinguishable cells ``` Two resolution numbers stand side by side, and they are not the same thing. The value of a single point is read from the position channel and there are 100 steps on the axis; it is the number of the first lesson. But for two points to be seen **as two separate points**, the distance between them has to exceed the width of the mark itself. The drawn point is not a mathematical point; it takes up a place on the page, and that place was taken as one fortieth of the axis. For this reason the plane is divided not into 100 times 100 but into **40 times 40** distinguishable cells, and 1,101 points will be put on 1,600 cells. #### As the Number of Points Grows ```python # scatter.py — as the number of points grows, how many points remain in a # separate cell. An overlapping point is not read back: the reader sees the # cell once. print(f"{'points':>7s}{'filled cells':>14s}{'hidden':>10s}{'hidden %':>12s}" f"{'densest cell':>16s}") for n in (100, 250, 500, 800, len(SHUFFLED)): s = defaultdict(int) for p in SHUFFLED[:n]: s[cell(p)] += 1 print(f"{n:7d}{len(s):14d}{n - len(s):10d}{100 * (n - len(s)) / n:11.1f}%" f"{max(s.values()):16d}") ``` ``` points filled cells hidden hidden % densest cell 100 85 15 15.0% 3 250 161 89 35.6% 5 500 232 268 53.6% 9 800 266 534 66.8% 12 1101 309 792 71.9% 16 ``` There are 1,600 cells in the plane and 1,101 points. There seems to be plenty of room, but the 1,101 points fall into only **309** cells, and **792 points**, that is 71.9%, have another point on top of them. The reason is that the points are not distributed evenly over the plane: where the relationship is, the points overplot, and precisely the region that carries the most information becomes the region that is least readable. In the densest cell 16 points overplot, and for the reader that cell is no different from a single point. How the column grows is also worth recording. At a hundred points the hidden rate is 15.0%; at two hundred and fifty points 35.6%; at five hundred 53.6%. While the number of points rises elevenfold, the number of filled cells rises only from 85 to 309, that is, threefold and a half. The information a scatter plot carries grows not with the number of points but with **the number of filled cells**, and this number has a ceiling: 1,600. #### The Heat Map and the Step Boundary Decision ```python # heatmap.py — the same pair set is divided into coarse intervals, the count # of every cell is turned into a color step and read back. The color is not a # name but a step number; the number of steps is 7, as in the first lesson. NX, NY, STEPS = 12, 10, 7 GX, GY = AXIS_X / NX, AXIS_Y / NY COUNT = defaultdict(int) for a, b in PAIRS: COUNT[(min(NX - 1, int(a / GX)), min(NY - 1, int(b / GY)))] += 1 ENC = max(COUNT.values()) def encode_color(c): return min(STEPS - 1, int(STEPS * c / ENC)) def decode_color(b): return (b + 0.5) * ENC / STEPS print(f"{NX}x{NY} intervals ({GX:.0f}x{GY:.0f} m3), densest cell {ENC} points, " f"{STEPS} color steps") for j in range(NY - 1, -1, -1): print(f"{(j + 1) * GY:5.0f} |" + "".join( str(encode_color(COUNT[(i, j)])) if COUNT[(i, j)] else "." for i in range(NX))) print(" +" + "-" * NX + f" x axis 0-{AXIS_X} m3") filled = [c for c in COUNT.values() if c > 0] levels_src = len(set(filled)) levels_read = len({encode_color(c) for c in filled}) rel = sum(abs(decode_color(encode_color(c)) - c) / c for c in filled) / len(filled) print(f"filled cells {len(filled)}, no point was lost: " f"{sum(filled)} = {len(PAIRS)}") print(f"the source has {levels_src} distinct count values, levels read back {levels_read}" f"/{STEPS}; mean relative read-back error {100 * rel:.1f}%") print("step occupancy: " + ", ".join( f"{b}:{sum(1 for c in filled if encode_color(c) == b)}" for b in range(STEPS))) # The second decision: the steps are divided not into equal intervals but into # an equal number of cells. The boundaries come from the sorted counts, and the # representative value of each step is the median of its own cells. _s = sorted(filled) BOUNDS = [_s[round(len(_s) * k / STEPS)] for k in range(1, STEPS)] def encode_rank(c): b = 0 while b < STEPS - 1 and c >= BOUNDS[b]: b += 1 return b REPRESENTATIVE = [] for b in range(STEPS): g = sorted(c for c in filled if encode_rank(c) == b) REPRESENTATIVE.append(g[len(g) // 2] if g else 0) rel2 = sum(abs(REPRESENTATIVE[encode_rank(c)] - c) / c for c in filled) / len(filled) print(f"equal cell count decision: levels read back " f"{len({encode_rank(c) for c in filled})}/{STEPS}, " f"mean relative error {100 * rel2:.1f}%") print("step occupancy: " + ", ".join( f"{b}:{sum(1 for c in filled if encode_rank(c) == b)}" for b in range(STEPS))) ``` ``` 12x10 intervals (5x5 m3), densest cell 140 points, 7 color steps 50 |............ 45 |........0... 40 |...000.000.. 35 |...00000000. 30 |..00121000.. 25 |..044310.... 20 |004641000... 15 |0142000..... 10 |00000....... 5 |.0.......... +------------ x axis 0-60 m3 filled cells 51, no point was lost: 1101 = 1101 the source has 28 distinct count values, levels read back 6/7; mean relative read-back error 280.9% step occupancy: 0:37, 1:5, 2:2, 3:1, 4:5, 5:0, 6:1 equal cell count decision: levels read back 6/7, mean relative error 15.7% step occupancy: 0:0, 1:13, 2:7, 3:8, 4:8, 5:8, 6:7 ``` The heat map does not lose what the scatter plot loses: all 1,101 points were counted in a cell and the total matches exactly. What it loses is something else — **the count itself**. There are 28 distinct count values in the source; the color step reduces them to seven, and in fact to six. The step occupancies are the direct evidence of CD24. The equal-interval division takes the 140 points of the densest cell as the measure and piles **37** of the 51 filled cells onto a single step, the lowest. The middle value of that step is 10 points; there are also single-point cells inside it. For this reason the mean relative read-back error comes out at 280.9%, and the reader sees most of the plane as a single level — the zeros lined up next to the zeros in the plane show this. The second decision encodes the same data with the same seven steps, only putting the boundaries somewhere else: the steps are divided not into equal intervals but into **an equal number of cells**. The occupancy is balanced and the mean relative error falls to 15.7%, to one eighteenth. It has a price, and it has to be written: a step series divided into an equal number of cells carries **not the magnitude but the order**. With the first decision the step difference of two cells tells how many points there are; with the second, only which one is more crowded. The same data, the same display, two defensible decisions, and an eighteenfold difference. #### What the Line Does Not Carry ```python # line.py — the third display: a line that gives a single y for every x. The # coefficient of the line was computed in M26/K05, and here it is neither # computed nor interpreted; the only thing measured is how much of the y at # that x a single y carries. The span is written in steps of the position # channel (the axis is divided into 100 steps). print(f"{'x interval':>12s}{'points':>7s}{'y smallest':>12s}{'y largest':>12s}" f"{'span steps':>17s}") steps_list = [] for i in range(NX): ys = [b for a, b in PAIRS if min(NX - 1, int(a / GX)) == i] if len(ys) < 20: continue n = round(RES_POSITION * (max(ys) - min(ys)) / AXIS_Y) steps_list.append(n) print(f"{i * GX:6.0f}-{(i + 1) * GX:<5.0f}{len(ys):7d}{min(ys):12.2f}" f"{max(ys):12.2f}{n:17d}") print(f"the line gives 1 step for every x; the cloud covers on average " f"{sum(steps_list) / len(steps_list):.1f} steps") ``` ``` x interval points y smallest y largest span steps 0-5 36 5.82 19.04 26 5-10 30 3.49 19.43 32 10-15 205 5.46 26.61 42 15-20 295 6.57 35.66 58 20-25 235 7.84 38.13 61 25-30 154 12.01 35.71 47 30-35 84 13.91 33.52 39 35-40 34 17.53 38.84 43 40-45 20 18.87 43.48 49 the line gives 1 step for every x; the cloud covers on average 44.1 steps ``` The third display is the one that takes up the least space and throws away the most. A line gives a single y for every x; that value is a single step of the position channel. In the same x interval the observations cover on average **44.1 steps**, that is, the number the line has read back takes up about a forty-fourth of the vertical span of the cloud. In the 15–20 m³ interval there are 295 subscribers and their means vary between 6.57 and 35.66 m³; the line puts a single number on this interval. This is not an objection to the line. The line exists in the first place to throw the span away, and if the report asks for a trend it is the right display. What is to be recorded is that what is thrown away is **countable**: when the span is not written next to the line, the reader takes a cloud of forty-four steps for one step. #### Summary - The relationship is encoded with two position channels; the value of a single point is read back with 100 steps, but the separation of two points depends on the mark width, and the plane comes down to 40 times 40 cells. - 1,101 points fall into only 309 cells: 792 points, 71.9%, overlap, and 16 points stand in the densest cell. The information a scatter plot carries grows not with the number of points but with the number of filled cells. - The heat map loses no point but loses the count: the 28 distinct count values in the source fall to six of the seven steps, 37 of the 51 filled cells land on a single step, and the relative error is 280.9%. - When the step boundary is set at an equal number of cells instead of equal intervals, the same error falls to 15.7%; the price is that the step now carries not the magnitude but the order. - A line that gives a single y for every x reduces a span of 44.1 steps on average to a single step; when the span is not written, the reader takes the cloud for a line. #### Next Step In four lessons four channels were measured: position, length, area, and color step — all on a single **snapshot**. The mean of five regions was at a moment, the distribution of north was the total of a period, the crosstab was a cross section, and both axes of the scatter plot belonged to the same time. When one of the axes is **time**, what the channel carries changes: the reader now reads not a value but a **change**, and the change itself is an unmeasured number. The next lesson moves to that axis and measures the read-back of the slope — what happens to the slope read back when the same series is encoded with two axis scales, and whether a trend can be read without separating the seasonal component. ### Time Series Visualization Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/time-series-visualization The same 36-period series is encoded at the same height with two axis scales: on the axis that starts from zero the reading resolution is 2.273 m³ and the mean read-back error per point is 0.540 m³, on the truncated axis 0.909 m³ and 0.246 m³. At a precision of 0.05 m³/period the slope is read back only from a window of 24 periods on the axis that starts from zero and from a window of 16 periods on the truncated axis. The seasonal component has a slope of its own and changes sign with the start of the window: the same 24-period length gives -0.2239, -0.0487 and -0.2260 m³/period in three separate places, while the trend put into the fictional series is -0.140. The previous four lessons encoded a number with four channels and measured what each reads back, all assuming a single snapshot. When the axis itself is **time**, the reader reads back not a value but the **change** of the value, and what is read is not a length but a **slope**. The measure of this lesson is the read-back of the slope, and two questions are asked: what happens to the slope read back when the same series is encoded at the same height with two axis spans, and whether the trend can be read back without separating the seasonal component. The resampling decision was measured in the dataframes topic of the Data Analysis with Python course and is not repeated; the series is taken as given and only its display is measured. - **CD26. What is read on the time axis is the slope.** The position difference between two points divided by the period difference gives the slope; the ratio of the length channel is not read here. - **CD27. The reading resolution is one row.** The vertical extent of the display is divided into a finite number of distinguishable positions; nothing finer than one row can be told apart. The resolution is the axis span divided by the number of rows, and every run prints it. - **CD28. The read-back error of the slope shrinks with the window length.** A slope read from the two ends errs by at most the resolution divided by the window length; in a short window the resolution swallows the slope. - **CD29. The seasonal component has a slope of its own.** The component has a zero mean over the whole series, but in a finite window its least squares slope is not zero and changes sign with where the window starts. The reader reads a single slope, the sum of two components. - **CD30. The series is fictional.** The three measured periods of the network cannot be extended to 36 periods; the series was derived with the same seed using the north region mean as the base, and its trend and seasonal amplitude are written in the code. #### When the Axis Is a Time ```python # time_series.py — MODEL: the measurement network of K03-K05 is # regenerated with the same seed (3,199 rows, 1,329 subscribers, five regions). # The 36-period series is also FICTIONAL: the three measured periods of the # network cannot be extended to 36; the series was derived with the same seed # by taking the north mean as the base. The trend and the seasonal amplitude # put into it are constant. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() north = [o["consumption_m3"] for o in READINGS if o["region"] == "north"] START = sum(north) / len(north) N, TREND, AMPLITUDE, SEASON = 36, -0.140, 2.20, 12 _g = rng(SEED + 505) SEASONAL = [AMPLITUDE * math.sin(2 * math.pi * t / SEASON) for t in range(N)] SERIES = [round(START + TREND * t + SEASONAL[t] + (_g() + _g() + _g() - 1.5) * 0.60, 3) for t in range(N)] def slope(v): """Least squares slope, unit: m3 / period.""" n = len(v) mt, mv = (n - 1) / 2, sum(v) / n return (sum((i - mt) * (x - mv) for i, x in enumerate(v)) / sum((i - mt) ** 2 for i in range(n))) def encode(v, low, high, rows): """Encoder: value -> position. Gives the exact row number on the grid.""" return max(0, min(rows - 1, round((v - low) / (high - low) * (rows - 1)))) def decode(k, low, high, rows): """Decoder: position -> value. The reader reads the value of the row itself.""" return low + k / (rows - 1) * (high - low) def draw(v, low, high, rows): pos = [encode(x, low, high, rows) for x in v] for s in range(rows - 1, -1, -1): print(f"{decode(s, low, high, rows):5.1f} |" + "".join("*" if k == s else " " for k in pos)) print(" +" + "-" * len(v)) return pos print(f"north region {len(north)} readings, mean {START:.3f} m3") print(f"fictional series: {N} periods, trend put in {TREND:+.3f} m3/period, " f"season length {SEASON} periods, amplitude {AMPLITUDE:.2f} m3") print(f"series minimum {min(SERIES):.2f}, maximum {max(SERIES):.2f} m3") print(f"series own slope {slope(SERIES):+.4f} m3/period") ``` ``` north region 921 readings, mean 20.210 m3 fictional series: 36 periods, trend put in -0.140 m3/period, season length 12 periods, amplitude 2.20 m3 series minimum 13.00, maximum 22.02 m3 series own slope -0.1797 m3/period ``` The last two lines carry the tension of the lesson. The trend put into the fictional series is -0.140 m³ per period, written in the code; the slope of the series itself comes out as -0.1797. The 0.0397 difference comes not from the noise but from the slope of the seasonal component itself, printed separately in the third section. It appears before any display is built, at the level of numbers alone. #### The Same Series, Two Axis Scales ```python # axis.py — the same series is encoded at the same height with two axis # scales. The reading resolution is one row: the reader cannot distinguish a # position finer than a row on the grid. ROWS, PRECISION = 12, 0.05 SOURCE = slope(SERIES) result = {} for name, low, high in (("zero-based", 0.0, 25.0), ("truncated", 13.0, 23.0)): c = (high - low) / (ROWS - 1) print(f"{name} axis: {low:.0f}-{high:.0f} m3, {ROWS} rows, " f"reading resolution {c:.3f} m3/row") pos = draw(SERIES, low, high, ROWS) back = [decode(k, low, high, ROWS) for k in pos] err = sum(abs(back[i] - SERIES[i]) for i in range(N)) / N print(f"mean read-back error of {N} points {err:.3f} m3; " f"the series uses {len(set(pos))} of the {ROWS} rows") result[name] = back print("") print(f"slope window zero-based truncated (window count) " f"precision {PRECISION:.2f} m3/period") for w in (4, 6, 9, 12, 15, 18, 24): count = [] for name in ("zero-based", "truncated"): g = result[name] count.append(sum(1 for i in range(N - w + 1) if abs(slope(g[i:i + w]) - slope(SERIES[i:i + w])) <= PRECISION)) print(f"{w:14d} {count[0]:9d} {count[1]:10d} {N - w + 1:18d}") for name in ("zero-based", "truncated"): g = result[name] full = [w for w in range(2, N + 1) if all(abs(slope(g[i:i + w]) - slope(SERIES[i:i + w])) <= PRECISION for i in range(N - w + 1))] shortest = next(w for w in full if all(v in full for v in range(w, N + 1))) print(f"on the {name} axis the slope is read back everywhere, at {PRECISION:.2f} " f"precision, only from a window of {shortest} periods") ``` ``` zero-based axis: 0-25 m3, 12 rows, reading resolution 2.273 m3/row 25.0 | 22.7 | ** * 20.5 |* * * *** 18.2 | ** *** ** ****** 15.9 | *** ***** ** 13.6 | **** 11.4 | 9.1 | 6.8 | 4.5 | 2.3 | 0.0 | +------------------------------------ mean read-back error of 36 points 0.540 m3; the series uses 5 of the 12 rows truncated axis: 13-23 m3, 12 rows, reading resolution 0.909 m3/row 23.0 | 22.1 | * 21.2 | * ** * 20.3 |* * * * 19.4 | * * * * 18.5 | * * * 17.5 | * * * ** 16.6 | *** * * 15.7 | * ** ** 14.8 | * 13.9 | * ** 13.0 | * +------------------------------------ mean read-back error of 36 points 0.246 m3; the series uses 11 of the 12 rows slope window zero-based truncated (window count) precision 0.05 m3/period 4 4 13 33 6 3 19 31 9 13 21 28 12 13 22 25 15 15 20 22 18 14 19 19 24 13 13 13 on the zero-based axis the slope is read back everywhere, at 0.05 precision, only from a window of 24 periods on the truncated axis the slope is read back everywhere, at 0.05 precision, only from a window of 16 periods ``` The two displays encode the same 36 numbers at the same height of 12 rows; only the axis span differs. On the axis that starts from zero the series uses just **5** of the 12 rows, on the truncated axis **11**. The mean read-back error per point is the numerical counterpart: 0.540 m³ against 0.246 m³. No number changed; the same numbers sat on a finer grid. The second table is the real measure. The slope written into the report is wanted correct to a precision of 0.05 m³/period, a threshold chosen because it is a tenth of the total change over three years. In a four-period window the axis that starts from zero hits the precision in only 4 of 33 windows, the truncated axis in 13; at six periods the difference is sharpest, 3 against 19. Hitting the threshold in every window takes **24 periods** on the axis that starts from zero and **16 periods** on the truncated axis. The basic charts lesson of the Analysis with Spreadsheets course measured axis truncation on the length channel, showing it detaches the bar ratio from the data. Here the same operation works in reverse: the length channel reads a ratio to the baseline, so moving the baseline breaks the ratio; the position channel reads the **difference** between two positions, independent of the baseline. The same decision taking the opposite sign in two channels shows that what decides is not the decision but the channel. #### The Season's Own Slope ```python # season.py — the seasonal component has a slope of its own, and it changes # sign according to where the window starts. step = [SERIES[i + 1] - SERIES[i] for i in range(N - 1)] print(f"of {len(step)} neighboring steps, {sum(1 for a in step if a > 0)} have a " f"sign opposite to the trend") print(f"{'window':12s}{'series slope':>15s}{'season slope':>16s}{'trend share':>13s}") for start in (0, 6, 12): p = slice(start, start + 24) print(f"{start:2d}-{start + 24:<2d} period {slope(SERIES[p]):+14.4f} " f"{slope(SEASONAL[p]):+15.4f} {TREND:+12.3f}") def smooth(v, p): """Centered moving average; for an even length the end values count as half.""" y = p // 2 return [(sum(v[i - y + 1:i + y]) + (v[i - y] + v[i + y]) / 2) / p for i in range(y, len(v) - y)] flat = smooth(SERIES, SEASON) flat_step = [flat[i + 1] - flat[i] for i in range(len(flat) - 1)] print(f"after the season is separated, {len(flat)} periods remain; " f"{sum(1 for a in flat_step if a > 0)} of the {len(flat_step)} steps are reversed") print(f"smoothed series slope {slope(flat):+.4f}, raw slope of the same window " f"{slope(SERIES[6:30]):+.4f}, trend put into the fictional series {TREND:+.3f}") ``` ``` of 35 neighboring steps, 17 have a sign opposite to the trend window series slope season slope trend share 0-24 period -0.2239 -0.0857 -0.140 6-30 period -0.0487 +0.0857 -0.140 12-36 period -0.2260 -0.0857 -0.140 after the season is separated, 24 periods remain; 0 of the 23 steps are reversed smoothed series slope -0.1347, raw slope of the same window -0.0487, trend put into the fictional series -0.140 ``` The first line gives the display's most misleading side: 17 of the 35 neighboring steps point against the trend. A reader who reads direction from two neighboring periods flips a coin. The middle table separates the cause. All three windows are 24 periods long and carry the same trend, but the slope of the series comes out as -0.2239, -0.0487 and -0.2260. The third column gives the source of the difference: the least squares slope of the seasonal component itself is -0.0857 in the window that starts from period 0 and **+0.0857** in the one that starts from period 6. The component has a zero mean over the whole series yet contributes to the slope in a finite window, and its sign depends on where the window starts within the season. This is a property of the series, not of the display; no axis setting can correct it, so the separation must come before the display. The last two lines are its result: with a moving average centered at the season length, none of the 23 steps keeps a reversed sign, and the slope read back becomes -0.1347. The raw slope of the same window was -0.0487; smoothing removed a deviation of 0.0913. The price is 12 periods, six dropped from each end of the series. #### Resolution and Season Bind Together ```python # both.py — the resolution and the season bind not separately but together. print("window axis only season only both together") trunc = result["truncated"] td = smooth(trunc, SEASON) sd = smooth(SERIES, SEASON) for w in (4, 6, 9, 12, 18): a = sum(1 for i in range(N - w + 1) if abs(slope(trunc[i:i + w]) - TREND) <= PRECISION) / (N - w + 1) b = sum(1 for i in range(len(sd) - w + 1) if abs(slope(sd[i:i + w]) - TREND) <= PRECISION) / (len(sd) - w + 1) c = sum(1 for i in range(len(td) - w + 1) if abs(slope(td[i:i + w]) - TREND) <= PRECISION) / (len(td) - w + 1) print(f"{w:7d} {a * 100:14.1f}% {b * 100:15.1f}% {c * 100:14.1f}%") print(f"trend in the source {TREND:+.3f} m3/period; the rates above are how often " f"the window gives this number at {PRECISION:.2f} precision") ``` ``` window axis only season only both together 4 3.0% 100.0% 85.7% 6 0.0% 100.0% 89.5% 9 7.1% 100.0% 100.0% 12 4.0% 100.0% 100.0% 18 100.0% 100.0% 100.0% trend in the source -0.140 m3/period; the rates above are how often the window gives this number at 0.05 precision ``` The previous table compared the slope read back with **the slope of the window itself**; this one compares it with the trend put into the fictional series, and its columns correspond to three separate corrections. The first column shows the insufficiency of the axis setting: the truncated axis does not give the trend in **any** window of six periods, and in a nine-period window gives it in two of 28 windows. The axis resolution corrects the positions of the points, but not the share the season adds to the slope. The second column shows the opposite: when the season is separated, even a four-period window hits the precision. The third column writes the price of the two together — the series encoded on the truncated axis and then smoothed starts at 85.7% and gives 100% from nine periods on. The remaining shortfall comes from quantization: every point is rounded to the row grid, and in a short window this rounding reaches the slope even after smoothing. The decision rule comes from here: a time series display **cannot be reported** without the **reading resolution of the axis** and the **window length over which the slope is read** written next to it. If either is missing, the slope the reader sees is not the trend in the data. #### Summary - What is read on the time axis is not length but slope; the reading resolution is one row, written as the axis span divided by the number of rows. - With the same series at the same height and two axis spans, the mean read-back error per point is 0.540 m³ and 0.246 m³; the axis that starts from zero uses 5 of the 12 rows. - At a precision of 0.05 m³/period the slope is read back in every window only from 24 periods on the axis that starts from zero and 16 periods on the truncated axis. - The seasonal component's own slope changes sign between -0.0857 and +0.0857 with the start of the window; the same 24-period length gives -0.2239, -0.0487 and -0.2260 m³/period in three places. - When the season is separated with a centered moving average, none of the 23 steps is left reversed and the slope read back becomes -0.1347; the price is 12 periods, and axis truncation cannot do this alone. #### Next Step This lesson turned the axis into time and measured that the channel carries change. Every reading of the measurement network is also taken at a place, a two-dimensional area. The next lesson models the regions as a grid and measures that a painted-area display reads back not a value but an **area**: the small value of a large region suppresses the large value of a small region, and the error is counted in proportion to the number of cells the region covers. Printing the same grid on a plane stretches it, and the stretching moves the number read back a third time. ### Geographic Visualization Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/geographic-visualization Five regions are encoded on a 12x12 coverage grid with five color steps, and what the reader reads back is not the step but the area the step covers: the correct mean weighted by the number of readings is 19.713 m³, while the read-back weighted by area gives 18.484 m³, a difference of -1.228 m³. West carries 14.9% of the readings and 25% of the area, so it gets a weight factor of 1.680. Projection stretching suppresses two regions of equal area by a factor of 1.237 relative to each other. When the area share is made equal to the read share, the error falls to 0.396 m³, at the price of 93 cells changing region and the pairs of neighboring regions dropping from 8 to 4. The previous lesson turned the axis into time and measured that what the channel carries is change. Every reading of the measurement network is also taken at a place, and that place is not a one-dimensional axis but a two-dimensional area. In a display built on an area, the value is turned into a color step and the reader reads the step back. But what enters the reader's eye is not the step but **the area the step covers**. This is the measure of this lesson: the difference between the number read back from a choropleth display and the number that exists in the data, and how this difference is proportional to the area of the regions. How many distinct levels the color step channel gets read back was measured in the relationship visualization lesson; here the number of steps is held fixed at five and the variable is the area. No real place, map product or projection name is written; the coverage area is modeled as a grid. - **CD31. The coverage grid is fictional.** The regions of the network are divided into a grid of 12x12 cells; each cell is a coverage unit, and the division is written in the code. - **CD32. In a choropleth display the channel is the color step.** The encoder maps the value to one of five equal-width slices and the decoder maps it to the middle of the slice; the reading resolution is the slice width, and because the reader cannot distinguish the position within the slice, this number gives the share that cannot be read back. - **CD33. The read-back is weighted by area.** The mean read at a glance is the mean of the region means weighted by the number of cells. The correct number is the one weighted by the number of readings. - **CD34. The weight factor is the region's area share divided by its read share.** If the factor is greater than one, the region is read more than it deserves; if it is smaller, less than it deserves. - **CD35. Projection stretching does not print equal areas equal.** When a curved surface is printed onto a rectangular grid, the rows are not printed at equal width; the stretch factor is a function of the row number in the model and is written in the code. - **CD36. The area share can be corrected, the adjacency cannot.** When the cells are distributed in proportion to the read share, the area weight disappears, but the information about which region neighbors which on the grid is spoiled, and the damage is measured by the number of pairs. #### The Grid and the Color Step ```python # grid.py — MODEL: the measurement network of K03-K05 is generated # with the same seed. The coverage area of the network is also FICTIONAL: it is # a 12x12 grid of cells and every cell is a coverage unit. No real place, map # or projection is mentioned. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() NAMES = [b[0] for b in REGION] COUNT = {a: sum(1 for o in READINGS if o["region"] == a) for a in NAMES} VALUE = {a: sum(o["consumption_m3"] for o in READINGS if o["region"] == a) / COUNT[a] for a in NAMES} # Coverage grid: 12x12, every cell belongs to a region. A fictional division. SIZE, STEPS = 12, 5 def cell_region(i, j): if i < 3: return "north" if j < 4: return "west" if i >= 8: return "south" if j >= 9: return "east" return "central" GRID = [[cell_region(i, j) for j in range(SIZE)] for i in range(SIZE)] AREA = {a: sum(r.count(a) for r in GRID) for a in NAMES} LOW, HIGH = min(VALUE.values()), max(VALUE.values()) WIDTH = (HIGH - LOW) / STEPS def encode(v): """Encoder: value -> color step. STEPS slices of equal width.""" return min(STEPS - 1, int((v - LOW) / WIDTH)) def decode(b): """Decoder: color step -> value. The reader reads the middle of the slice.""" return LOW + (b + 0.5) * WIDTH print(f"grid {SIZE}x{SIZE} = {SIZE * SIZE} cells, {STEPS} color steps, " f"reading resolution {WIDTH:.3f} m3/step") print(f"{'region':7s} {'area(cells)':>11s} {'count':>6s} {'mean':>9s} {'step':>8s} " f"{'read-back':>12s} {'error':>6s}") for a in NAMES: b = encode(VALUE[a]) print(f"{a:7s} {AREA[a]:11d} {COUNT[a]:6d} {VALUE[a]:9.3f} {b:8d} " f"{decode(b):12.3f} {decode(b) - VALUE[a]:+6.3f}") for i in range(SIZE): print(" " + "".join(str(encode(VALUE[GRID[i][j]])) for j in range(SIZE))) ``` ``` grid 12x12 = 144 cells, 5 color steps, reading resolution 2.179 m3/step region area(cells) count mean step read-back error north 36 921 20.210 2 19.226 -0.985 south 32 645 16.602 1 17.046 +0.445 east 15 557 24.674 4 23.584 -1.090 west 36 476 13.778 0 14.867 +1.090 central 25 600 22.396 3 21.405 -0.991 222222222222 222222222222 222222222222 000033333444 000033333444 000033333444 000033333444 000033333444 000011111111 000011111111 000011111111 000011111111 ``` The step column is the whole of the channel: five regions fell on five different steps and each one can be told apart. The read-back error per region is between -1.090 and +1.090 m³, that is, at most half the slice width. This is the known and bounded price of the color step channel. The grid at the bottom shows the real problem. The step of east is 4, the highest value of the grid, but it covers only 15 cells on the page. The step of west is 0 and it covers 36 cells. Someone looking at the same grid gets the impression "this network consumes little"; the step that covers the most room on the page is the lowest one. The error per region reports this in no way, because the error is measured region by region and the reader does not look region by region. #### Area Weight ```python # weight.py — what the reader reads back at a glance is not the step but the # area the step covers. Two means are put side by side. TOTAL = sum(COUNT.values()) correct = sum(VALUE[a] * COUNT[a] for a in NAMES) / TOTAL glance = sum(decode(encode(VALUE[a])) * AREA[a] for a in NAMES) / (SIZE * SIZE) plain = sum(decode(encode(VALUE[a])) for a in NAMES) / len(NAMES) print(f"mean weighted by readings {correct:.3f} m3 ({TOTAL} readings)") print(f"plain mean per region {plain:.3f} m3 ({len(NAMES)} regions)") print(f"read-back weighted by area {glance:.3f} m3 ({SIZE * SIZE} cells)") print(f"difference opened by area weight {glance - correct:+.3f} m3") print(f"{'region':7s}{'read share':>11s}{'area share':>11s}{'weight factor':>17s}") for a in NAMES: rs, ash = COUNT[a] / TOTAL, AREA[a] / (SIZE * SIZE) print(f"{a:7s} {rs * 100:9.1f}% {ash * 100:9.1f}% {ash / rs:16.3f}") ``` ``` mean weighted by readings 19.713 m3 (3199 readings) plain mean per region 19.226 m3 (5 regions) read-back weighted by area 18.484 m3 (144 cells) difference opened by area weight -1.228 m3 region read share area share weight factor north 28.8% 25.0% 0.868 south 20.2% 22.2% 1.102 east 17.4% 10.4% 0.598 west 14.9% 25.0% 1.680 central 18.8% 17.4% 0.926 ``` Three means, from the same data and none of them miscalculated. The mean of the 3,199 readings is 19.713 m³; the plain mean per region is 19.226; the one read from the choropleth display is 18.484. The difference opened by the area weight is **-1.228 m³**, and not a single reading changed. The table at the bottom writes where the difference comes from, region by region. West carries 14.9% of the readings but covers 25% of the area; its weight factor is **1.680**, that is, on the page it speaks about twice as loudly as its real share. East, on the contrary, with a factor of 0.598, speaks at a little more than half of its share. West having the lowest mean and east the highest is a coincidence, but when two coincidences work in the same direction the number read back shifts to one side. This is not a flaw independent of the choice of channel; it is **the definition of the area channel**. The color step is painted onto the area, and the area is already fixed and has nothing to do with the data. The display applies a weighting the designer did not choose, and writes that weight nowhere. #### Projection Stretching ```python # stretch.py — when a curved surface is printed onto a rectangular grid, the # rows are not printed at equal width. The stretch factor is a MODEL: upper # rows are printed wider. CURVE, REF = 0.07, SIZE - 1 def stretch(i): return 1.0 / math.cos(CURVE * (REF - i)) PRINTED = {a: sum(stretch(i) for i in range(SIZE) for j in range(SIZE) if GRID[i][j] == a) for a in NAMES} TOTAL_AREA = sum(PRINTED.values()) stretched = sum(decode(encode(VALUE[a])) * PRINTED[a] for a in NAMES) / TOTAL_AREA print(f"stretch factor in the top row {stretch(0):.3f}, in the bottom row " f"{stretch(SIZE - 1):.3f}") print(f"{'region':7s}{'true area':>12s}{'printed area':>14s}{'growth':>8s}") for a in NAMES: print(f"{a:7s} {AREA[a]:11d} {PRINTED[a]:13.2f} " f"{PRINTED[a] / AREA[a] - 1:+7.1%}") print(f"north and west have equal areas ({AREA['north']} cells) but their printed " f"areas differ by a factor of {PRINTED['north'] / PRINTED['west']:.3f}") print(f"area-weighted read-back before stretching {glance:.3f} m3, " f"after {stretched:.3f} m3") print(f"distance to the correct number: {abs(glance - correct):.3f} -> " f"{abs(stretched - correct):.3f} m3") ``` ``` stretch factor in the top row 1.393, in the bottom row 1.000 region true area printed area growth north 36 47.26 +31.3% south 32 32.28 +0.9% east 15 16.54 +10.3% west 36 38.19 +6.1% central 25 27.57 +10.3% north and west have equal areas (36 cells) but their printed areas differ by a factor of 1.237 area-weighted read-back before stretching 18.484 m3, after 18.579 m3 distance to the correct number: 1.228 -> 1.133 m3 ``` Stretching is a second distortion on top of the area weight, and this time it is not even the designer's decision: it is the unavoidable price of printing a curved surface onto a flat grid. In the model the top row is printed 1.393 times as wide as the bottom row. The fifth line is the clearest statement of the measure. North and west are **of equal area**, both 36 cells. When printed they come to 47.26 and 38.19, that is, two regions of the same size look **1.237 times** different in size. A reader has no basis whatsoever for comparing the weight of the two regions. The last two lines must be read carefully. In this fictional setup stretching **lowers** the total error from 1.228 to 1.133 m³. This is not a correction but a coincidence: the region that stretches the most is north, and because the read-back value of north is close to the correct number, it has pulled the mean upward. The sign of the stretching depends on where the regions stand on the grid and is not under the designer's control. Its lowering the error is as accidental as its raising it; what has to be measured is not the total but the deviation of the printed area per region. #### Correcting the Area Share ```python # cartogram.py — the cells are redistributed in proportion to the read share. # The area weight is corrected, the adjacency structure of the grid is spoiled. quota = {a: COUNT[a] * SIZE * SIZE / TOTAL for a in NAMES} share = {a: int(quota[a]) for a in NAMES} for a in sorted(NAMES, key=lambda x: quota[x] - int(quota[x]), reverse=True): if sum(share.values()) < SIZE * SIZE: share[a] += 1 stream = [a for a in NAMES for _ in range(share[a])] CARTO = [stream[i * SIZE:(i + 1) * SIZE] for i in range(SIZE)] NEW = {a: sum(r.count(a) for r in CARTO) for a in NAMES} cartoview = sum(decode(encode(VALUE[a])) * NEW[a] for a in NAMES) / (SIZE * SIZE) moved = sum(1 for i in range(SIZE) for j in range(SIZE) if GRID[i][j] != CARTO[i][j]) def adjacency(g): c = set() for i in range(SIZE): for j in range(SIZE): for di, dj in ((0, 1), (1, 0)): x, y = i + di, j + dj if x < SIZE and y < SIZE and g[i][j] != g[x][y]: c.add(tuple(sorted((g[i][j], g[x][y])))) return c k1, k2 = adjacency(GRID), adjacency(CARTO) print(f"{'region':7s}{'old area':>10s}{'cartogram area':>16s}{'read share':>12s}") for a in NAMES: print(f"{a:7s} {AREA[a]:9d} {NEW[a]:15d} {COUNT[a] / TOTAL * 100:10.1f}%") for i in range(SIZE): print(" " + "".join(str(encode(VALUE[CARTO[i][j]])) for j in range(SIZE))) print(f"area-weighted read-back {glance:.3f} -> {cartoview:.3f} m3, " f"correct number {correct:.3f}") print(f"error {abs(glance - correct):.3f} -> {abs(cartoview - correct):.3f} m3; " f"{moved} cells changed region") print(f"adjacent region pairs {len(k1)} -> {len(k2)}; lost " f"{len(k1 - k2)}, invented {len(k2 - k1)}") ``` ``` region old area cartogram area read share north 36 42 28.8% south 32 29 20.2% east 15 25 17.4% west 36 21 14.9% central 25 27 18.8% 222222222222 222222222222 222222222222 222222111111 111111111111 111111111114 444444444444 444444444444 000000000000 000000000333 333333333333 333333333333 area-weighted read-back 18.484 -> 19.317 m3, correct number 19.713 error 1.228 -> 0.396 m3; 93 cells changed region adjacent region pairs 8 -> 4; lost 6, invented 2 ``` The correction is simple: every region is given a number of cells proportional to its read share. The area of east goes from 15 up to 25, that of west from 36 down to 21. The read-back error falls from **1.228 to 0.396 m³**, that is, to a third. The remaining 0.396 comes not from the area weight but from the 2.179 m³ resolution of the color step, and cannot be closed by the area correction. The price is written with two numbers. The first is that **93** of the 144 cells changed region; two thirds of the grid is no longer where it started. The second is heavier: the original grid had **8** pairs of neighboring regions, the cartogram is left with **4**. Six adjacencies were lost and two adjacencies that do not exist were invented. The cartogram makes the correct number read back, but spoils the answer to the question "which region is adjacent to which"; if the reader assumes they can ask the second question from this display too, they fall into a new error. The decision rule comes from here. A display built on an area cannot be reported without two numbers: **the ratio of the region's area share to its read share** and **the resolution of the color step**. If the area share is corrected, the number of pairs of adjacency lost is also written, because the correction has made another question unreadable. #### Summary - In a choropleth display the channel is the color step; five steps give a reading resolution of 2.179 m³ and the read-back error per region is at most 1.090 m³. - The number the reader reads back at a glance is weighted by area: the correct mean of 3,199 readings is 19.713 m³, while the one read from the display comes out as 18.484 m³, a difference of -1.228 m³. - The weight factor is the area share divided by the read share; west gets a factor of 1.680 and east 0.598, and the display writes this weight nowhere. - Projection stretching prints north and west, which are of equal area, 1.237 times different; that it reduces the total error by 0.095 m³ in this fictional setup is a coincidence, and its sign is not under the designer's control. - When the area share is made equal to the read share, the error falls to 0.396 m³; the price is 93 cells changing places and the pairs of neighboring regions dropping from 8 to 4. #### Next Step None of the distortions measured in this lesson arose from a choice: the area is given on the grid, and the stretching is the price of printing onto a flat surface. Even so, the number read back slipped by 1.228 m³. The next lesson asks the same question from the opposite side and takes four distortions as a **family** — axis truncation, the area channel enlarging the ratio by its square, the dual axis and the uneven interval are applied to the same data in turn, and the read-back error of each is printed separately. All four arise from a defensible default behavior; what is to be shown is that misleading does not require intent. ### Misleading Visuals Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/misleading-visuals The true ratio of 1.2208 between east and north gives a bar ratio of 3.0769 when the axis is started from 18 m³, and a reader who takes length as proportional to value reads north as 8.019 m³ instead of 20.210. When the same ratio is encoded into a square, both sides are multiplied, so the area ratio comes out as 1.4905 and east is read as 30.123 m³. For two series with different units, nine defensible axis pairs give seven separate crossing periods. When 12 irregularly spaced periods are laid out at equal intervals, a slope of -0.2079 m³/period is read as -0.6635. All four arise from a defensible default behavior. The previous lesson measured a distortion without tying it to any choice: the area of the regions is given on the grid and the stretching is the price of printing onto a flat surface. Even so, the number read back slipped by 1.228 m³. This lesson widens the same question and treats four separate distortions as a single **family**. The common feature of the family is this: every member arises from the definition of the channel itself, none of them is a calculation error and none of them requires intent. How axis truncation distorts the ratio of bar lengths was measured in the basic charts lesson of the Analysis with Spreadsheets course, and that number is not repeated here. There the measure was **the legitimacy of the display** — is this chart acceptable at this scale. Here truncation is only one of the four members of the family and the first of four operations applied in turn to the same data. What is measured is the number each member makes the reader read back. - **CD37. The distortion arises from the definition of the channel.** The length channel is read relative to the base, the area channel grows in two dimensions at once, the position channel depends on the axis span. None of them spoils the calculation, all three spoil the read-back. - **CD38. The reader's decoder cannot be changed.** The reader takes length as proportional to value, area as proportional to value, and neighboring points as separated by an equal period interval. If the display falsifies this assumption, the error is written not to the display but to the reader. - **CD39. The area channel enlarges the ratio by its square.** If both sides of a shape are multiplied to "show a ratio as larger", the area ratio becomes the square of the ratio of values. - **CD40. A crossing on a dual axis has no counterpart in the data.** If the two series are in different units, the crossing period is determined by the choice of the two axis ranges, and although the choices are defensible they do not give a single number. - **CD41. When an uneven interval is laid out equally, the slope is measured in the number of steps.** The reader takes a step as a period; the enlargement factor is the period span divided by the number of steps. - **CD42. The series are fictional.** The consumption and reading count series of 36 periods were derived with the same seed, and that they are fictional is written in the code. #### Length: Axis Truncation ```python # misleading.py — MODEL: the measurement network of K03-K05 is # generated with the same seed. Both series of 36 periods are also FICTIONAL and # were derived with the same seed: the consumption mean (m3) and the number of # readings per period (count). import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() NAMES = [b[0] for b in REGION] VALUE = {a: sum(o["consumption_m3"] for o in READINGS if o["region"] == a) / sum(1 for o in READINGS if o["region"] == a) for a in NAMES} N = 36 _g = rng(SEED + 505) CONSUMPTION = [round(VALUE["north"] - 0.140 * t + 2.20 * math.sin(2 * math.pi * t / 12) + (_g() + _g() + _g() - 1.5) * 0.60, 3) for t in range(N)] _h = rng(SEED + 917) READ_COUNT = [round(1066 + 6.4 * t + 40 * math.sin(2 * math.pi * t / 12) + (_h() + _h() + _h() - 1.5) * 26) for t in range(N)] WIDTH = 40 def length(v, base, top): """Encoder: value -> length. The bar is rounded to a whole character.""" return max(0, round(WIDTH * (v - base) / (top - base))) def decode_length(u, base, top): """Decoder: length -> value. The reading resolution is one character.""" return base + u / WIDTH * (top - base) a, b = VALUE["east"], VALUE["north"] TRUE_RATIO = a / b print(f"east {a:.3f} m3, north {b:.3f} m3, true ratio {TRUE_RATIO:.4f}") for base in (0.0, 18.0): ua, ub = length(a, base, a), length(b, base, a) print(f"axis from {base:4.0f} m3: east |{'#' * ua}") print(f" north |{'#' * ub}") print(f" bar ratio {ua / ub:.4f}; a reader who takes length as proportional " f"to value reads north as {decode_length(ub, 0.0, a):.3f} m3") TRUNCATED = length(a, 18.0, a) / length(b, 18.0, a) print(f"truncation ratio {TRUE_RATIO:.4f} -> {TRUNCATED:.4f}, enlargement factor " f"{TRUNCATED / TRUE_RATIO:.3f}; read-back error at north " f"{decode_length(length(b, 18.0, a), 0.0, a) - b:+.3f} m3") ``` ``` east 24.674 m3, north 20.210 m3, true ratio 1.2208 axis from 0 m3: east |######################################## north |################################# bar ratio 1.2121; a reader who takes length as proportional to value reads north as 20.356 m3 axis from 18 m3: east |######################################## north |############# bar ratio 3.0769; a reader who takes length as proportional to value reads north as 8.019 m3 truncation ratio 1.2208 -> 3.0769, enlargement factor 2.520; read-back error at north -12.191 m3 ``` The two sets of bars carry the same two numbers. On the axis that starts from zero the ratio comes out as 1.2121; the true ratio is 1.2208, and the difference comes from the bars being rounded to whole characters, that is, from the reading resolution. On the axis that starts from 18 m³ the ratio of the same two numbers becomes **3.0769**. The real number is on the last line. A reader who takes length as proportional to value reads north from the truncated display as **8.019 m³**; the true value is 20.210 and the read-back error is **-12.191 m³**. Neither did the data change nor was a calculation done wrongly. Where this decision comes from matters. The means of the two regions are squeezed between 20.210 and 24.674, and on an axis that starts from zero the two bars look almost the same. "Make the difference visible" is a design requirement, and the previous course had treated it as a legitimate decision — since what is read on a line chart is the slope, the axis may be truncated there. When the same decision is applied to the length channel, it falsifies the reader's decoder. The decision is the same, the channel is different, and the result is an error of 12 m³. #### Area: The Square of the Ratio ```python # area.py — the decision to "show it twice as large" enlarges the shape in two # dimensions at once; the reader reads the area and the ratio returns squared. SIDE = 12 def encode_area(ratio): """Encoder: ratio -> area. Both sides are multiplied by the ratio.""" k = SIDE * ratio return k * k def decode_area(ar): """Decoder: area -> value. The reader takes the area as proportional to the value.""" return b * ar / (SIDE * SIDE) area_a, area_b = encode_area(TRUE_RATIO), encode_area(1.0) print(f"north square {SIDE}x{SIDE} = {area_b:.0f} area units") print(f"east square {SIDE * TRUE_RATIO:.2f}x{SIDE * TRUE_RATIO:.2f} = {area_a:.1f} area units") print(f"area ratio {area_a / area_b:.4f} = square of the true ratio {TRUE_RATIO ** 2:.4f}") print(f"east value read back {decode_area(area_a):.3f} m3, true {a:.3f} m3, " f"error {decode_area(area_a) - a:+.3f} m3") ``` ``` north square 12x12 = 144 area units east square 14.65x14.65 = 214.6 area units area ratio 1.4905 = square of the true ratio 1.4905 east value read back 30.123 m3, true 24.674 m3, error +5.449 m3 ``` The second member of the family does not touch the axis at all. To show that east is 1.2208 times larger than north, the square of east is scaled by 1.2208; the scaling is applied to both sides at once because the square must remain a square. The result is that the area ratio is **1.4905**, and this number is the square of the ratio of values. The reader sees the area, not the side. A reader who takes area as proportional to value reads east as **30.123 m³**; the true value is 24.674 and the read-back error is **+5.449 m³**. The enlargement factor is 1.2208, that is, as large as the ratio itself — the larger the ratio, the larger the distortion. This member differs from axis truncation in two respects. First, it cannot be corrected: making the ratio of the sides the square root of the value corrects the area, but then the ratio of the sides no longer belongs to the data, and someone who reads the side is mistaken. Second, the decision itself is invisible; the axis base is a number and can be written down, but "which dimension was scaled" stands nowhere in the display. #### Position: Dual Axis and Uneven Interval ```python # dual.py — two series in two separate units. When each axis is stretched to # its own range, the crossing period is determined by the choice of axis. ROWS = 12 def position(v, low, high): return max(0, min(ROWS - 1, round((v - low) / (high - low) * (ROWS - 1)))) def options(v): lo, hi = min(v), max(v) return {"from zero": (0.0, hi), "data range": (math.floor(lo), math.ceil(hi)), "wide": (math.floor(lo) - (hi - lo) * 0.5, math.ceil(hi))} oc, on = options(CONSUMPTION), options(READ_COUNT) print(f"{'consumption axis':18s}{'count axis':15s}{'crossing period':>15s}") crossing = set() for na, (a_lo, a_hi) in oc.items(): for nb, (b_lo, b_hi) in on.items(): k = next((t for t in range(N) if position(CONSUMPTION[t], a_lo, a_hi) <= position(READ_COUNT[t], b_lo, b_hi)), None) crossing.add(k) print(f"{na:16s} {nb:14s} {str(k):>15s}") print(f"{len(oc) * len(on)} defensible axis pairs give " f"{len(crossing)} different crossing periods") print(f"the units of the two series differ (m3 and count); the crossing period is " f"a number with no counterpart in the data") ``` ``` consumption axis count axis crossing period from zero from zero 5 from zero data range 25 from zero wide 19 data range from zero 0 data range data range 18 data range wide 8 wide from zero 0 wide data range 19 wide wide 17 9 defensible axis pairs give 7 different crossing periods the units of the two series differ (m3 and count); the crossing period is a number with no counterpart in the data ``` The dual axis is the hardest member of the family to defend, because its reason for existing is legitimate: the consumption mean is in m³, the number of readings per period is in count, and putting the two on a single axis flattens one series into a straight line. Stretching each series to its own range is the natural solution. The price is in the table. Nine defensible axis pairs give **seven separate crossing periods**, and among them are completely different answers such as period 0 and period 25. The sentence "in this period one overtook the other" that the reader draws from this display corresponds to nothing in the data; the impression that two different units are comparable is entirely a product of the axis choice. In the other three members the number read back had an error; here the number read back **has no source**. ```python # interval.py — if the periods on hand are laid out at equal intervals, the # slope is spoiled. HELD = [0, 1, 2, 3, 4, 5, 6, 12, 18, 24, 30, 35] def slope(xs, ys): n = len(xs) mx, my = sum(xs) / n, sum(ys) / n return (sum((xs[i] - mx) * (ys[i] - my) for i in range(n)) / sum((x - mx) ** 2 for x in xs)) y = [CONSUMPTION[t] for t in HELD] gaps = [HELD[i + 1] - HELD[i] for i in range(len(HELD) - 1)] correct = slope(HELD, y) drawn = slope(list(range(len(HELD))), y) print(f"{len(HELD)} periods on hand, spread from period {HELD[0]} to period " f"{HELD[-1]}; gaps {min(gaps)}-{max(gaps)} periods") print(f"true slope {correct:+.4f} m3/period ({HELD[-1] - HELD[0]} periods of spread)") print(f"slope read from the display drawn at equal intervals {drawn:+.4f} " f"m3/period ({len(HELD) - 1} steps), enlargement {drawn / correct:.2f} times") print(f"{'step':>4s}{'true gap':>15s}{'true slope':>13s}{'drawn slope':>14s}") for i in (0, 6, 7, len(gaps) - 1): print(f"{i:4d} {gaps[i]:14d} {(y[i + 1] - y[i]) / gaps[i]:+12.4f} " f"{y[i + 1] - y[i]:+13.4f}") TABLE = [("axis truncation", "length", f"{TRUE_RATIO:.4f}", f"{TRUNCATED:.4f}", f"{TRUNCATED / TRUE_RATIO:.2f}x", "axis base"), ("area enlargement", "area", f"{TRUE_RATIO:.4f}", f"{area_a / area_b:.4f}", f"{(area_a / area_b) / TRUE_RATIO:.2f}x", "whether side or area was scaled"), ("dual axis", "position", "none", f"{len(crossing)} separate periods", "-", "range of the two axes"), ("uneven interval", "position", f"{correct:.4f}", f"{drawn:.4f}", f"{drawn / correct:.2f}x", "period gaps")] print(f"{'distortion':18s} {'channel':8s} {'source':>8s} {'read back':>18s} {'growth':>9s} unwritten decision") for name, channel, src, back, grow, row in TABLE: print(f"{name:18s} {channel:8s} {src:>8s} {back:>18s} {grow:>9s} {row}") ``` ``` 12 periods on hand, spread from period 0 to period 35; gaps 1-6 periods true slope -0.2079 m3/period (35 periods of spread) slope read from the display drawn at equal intervals -0.6635 m3/period (11 steps), enlargement 3.19 times step true gap true slope drawn slope 0 1 +1.3370 +1.3370 6 6 -0.1895 -1.1370 7 6 -0.0655 -0.3930 10 5 -0.3822 -1.9110 distortion channel source read back growth unwritten decision axis truncation length 1.2208 3.0769 2.52x axis base area enlargement area 1.2208 1.4905 1.22x whether side or area was scaled dual axis position none 7 separate periods - range of the two axes uneven interval position -0.2079 -0.6635 3.19x period gaps ``` The fourth member is the one that appears most often and most unnoticed. There are 12 periods on hand and they are spread from period 0 to period 35; the gaps vary between 1 and 6 periods. The display lays them out in order and at equal intervals, because laying out what is on hand in order is the default behavior. The slope is read as **-0.6635** instead of -0.2079 m³/period, **3.19 times** larger. The factor is close to the period span divided by the number of steps: 35 divided by 11, that is, 3.18. The step table shows how this comes about step by step — at step 0 the true gap is one period and the two numbers coincide, at step 6 the gap rises to six periods and the drop of -0.1895 per period is read in the display as -1.1370. The last table puts the four members side by side. Its last column is what the four have in common: **in each of them, the information that would correct the read-back is left outside the display**. The axis base is a number and can be written down; so can the period gaps. "Which dimension was scaled" and "what is the range of the two axes" are known only where the display is made and never reach the reader. All four arose from a defensible default: make the difference visible, enlarge the shape by the ratio, stretch the two series to their own ranges, lay out what is on hand in order. Misleading required no intent in any of them. #### Summary - The four distortions are a single family: they arise from the definition of the channel itself, do not spoil the calculation, and spoil the read-back. - When the axis is started from 18 m³, the ratio of 1.2208 is read as 3.0769, and a reader who takes length as proportional to value reads north as 8.019 m³ instead of 20.210, an error of -12.191 m³. - When the same ratio is encoded into a square, both sides are multiplied at once, so the area ratio is 1.4905, that is, the square of the ratio; east is read as 30.123 m³ and the error is +5.449 m³. - In two series with different units, nine defensible axis pairs give seven separate crossing periods; the number read back has no counterpart in the source. - When 12 irregularly spaced periods are laid out equally, a slope of -0.2079 m³/period is read as -0.6635; the factor is close to the period span divided by the number of steps, 35 divided by 11, that is, 3.18. - In all four, the information that would correct the read-back is left outside the display, and all four arise from a defensible default behavior. #### Next Step This lesson showed that distortion does not require intent; in every member the display had been built correctly and not a single number had been calculated wrongly. One assumption still stands untouched: that the reader is able to read the channel. The next lesson measures this. How many series remain indistinguishable from one another when color is the only encoding channel is counted, and how many this number falls to with **redundant encoding**, made up of shape, direct labeling and ordering, is measured. The contrast ratio calculation and the color system are the business of the interface courses and are not repeated; what is counted here is only what is specific to the data. ### Accessible Charts Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/accessible-charts When eight series are encoded with a default palette made of two hue families and four lightness steps, all eight are distinguished by the full decoder, while a decoder that does not read the hue distinction is left with only four. When the shape channel is distributed in sequence, the number stays at four, because the shapes start over after four series and the color collision also occurs four series apart; when the shape is distributed by collision, it rises to eight. A legend box asks for 16 transitions, an ordered legend 6, direct labeling 0. Of nine lightness steps, five are distinguished at an interval of two, and with four shapes the ceiling of a single display comes out as 20 series. Over seven lessons what was measured was the number the channel carries, and each time it was assumed that the reader **can read** the channel. This lesson measures that assumption. If there is more than one series in a display, the first question is not whether the values of the series are read correctly; the first question is which series is which. If a series cannot be distinguished from its neighbor, its value is also counted as not reported. The contrast ratio calculation, color systems and accessibility criteria were established in the Fundamentals of Interface Design course and in the accessibility courses; none of them is repeated here and no color system name is written. The question here is specific to the data and is measured by a single number: **the number of distinguishable series**. When color is the only encoding channel, what is this number, and to what does redundant encoding raise it. - **CD43. Color is not a name but two numbers.** In the model, color is made of a hue family and a lightness step. The encoder assigns a pair to a series; no color name or system is mentioned. - **CD44. The reading resolution is two steps.** Two lightness steps cannot be distinguished unless they differ by at least two steps, because in the display the series are thin lines and stand far from one another; this is not the distinction of two wide surfaces placed side by side. - **CD45. Two decoders run.** The full decoder reads both the hue family and the lightness step. The reduced decoder does not read the hue distinction and sees only the lightness step; both are applied to the same encoding. - **CD46. The number of distinguishable series is the number of clusters.** Series that cannot be told apart from one another are gathered into a single cluster; the number of series read back is the number of clusters and is smaller than the total of the series. - **CD47. The place of redundant encoding determines the result.** Adding a second channel is not enough by itself; the channel has to change on the axis on which the color collides. - **CD48. A single display has a ceiling.** The ceiling is the product of the number of distinguishable steps and the number of shapes; above it, the number of series cannot be read back without splitting the display. #### When Color Is the Only Channel ```python # series.py — MODEL: the measurement network of K03-K05 is generated # with the same seed. The eight series are subsets of the same 3,199 readings: # five regions and three tariff tiers. Color is not a name but two numbers: # hue family and lightness step. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] TARIFF = [10, 25, 40] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _kept, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _kept).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {a["subscriber_id"] for a in _kept} READINGS = [] for a in _kept + _dropped: r = rng(SEED + 7000 + a["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if a["subscriber_id"] not in VALID: r() continue READINGS.append({"subscriber_id": a["subscriber_id"], "region": a["region"], "period": PERIODS[d], "consumption_m3": t}) r() def tier(v): return 1 + sum(1 for t in TARIFF if v >= t) def average(f): v = [o["consumption_m3"] for o in READINGS if f(o)] return sum(v) / len(v), len(v) SERIES = [] for name, _, _ in REGION: m, n = average(lambda o, x=name: o["region"] == x) SERIES.append((name, m, n)) for tr in (2, 3, 4): m, n = average(lambda o, x=tr: tier(o["consumption_m3"]) == x) SERIES.append((f"tier{tr}", m, n)) # Color channel: hue family (0/1) and lightness step (0-8). The palette picks # four steps from each of two families; this is a natural default distribution # for eight series. LEVELS, SEPARATION, SHAPES = 9, 2, 4 PALETTE = [(i // 4, [1, 3, 5, 7][i % 4]) for i in range(len(SERIES))] def full_decoder(x, y): """Are two series distinguishable: the hue differs, or the step difference is SEPARATION.""" return x[0] != y[0] or abs(x[1] - y[1]) >= SEPARATION def reduced_decoder(x, y): """The decoder that does not read the hue distinction sees only the lightness step.""" return abs(x[1] - y[1]) >= SEPARATION def distinguishable(code, decoder): """Series that cannot be told apart form one cluster; the number of clusters is the series read back.""" root = list(range(len(code))) def find(i): while root[i] != i: root[i] = root[root[i]] i = root[i] return i for i in range(len(code)): for j in range(i + 1, len(code)): if not decoder(code[i], code[j]): root[find(i)] = find(j) return len({find(i) for i in range(len(code))}) print(f"{len(SERIES)} series, all subsets of the same {len(READINGS)} readings") print(f"{'series':11s} {'count':>6s} {'mean':>9s} {'hue':>4s} {'light':>8s}") for (name, m, n), (t, light) in zip(SERIES, PALETTE): print(f"{name:11s} {n:6d} {m:9.3f} {t:4d} {light:8d}") print(f"reading resolution: two lightness steps cannot be distinguished unless " f"they differ by at least {SEPARATION} steps") print(f"distinguishable series with the full decoder: " f"{distinguishable(PALETTE, full_decoder)}/{len(SERIES)}") print(f"with the decoder that does not read the hue distinction: " f"{distinguishable(PALETTE, reduced_decoder)}/{len(SERIES)}") ``` ``` 8 series, all subsets of the same 3199 readings series count mean hue light north 921 20.210 0 1 south 645 16.602 0 3 east 557 24.674 0 5 west 476 13.778 0 7 central 600 22.396 1 1 tier2 2209 17.506 1 3 tier3 692 30.009 1 5 tier4 58 44.171 1 7 reading resolution: two lightness steps cannot be distinguished unless they differ by at least 2 steps distinguishable series with the full decoder: 8/8 with the decoder that does not read the hue distinction: 4/8 ``` The palette looks flawless. Each of the eight series has received a separate color pair, no pair has been repeated, and the full decoder distinguishes all eight. This is the result seen by the person who chose the palette. The last line says what happens to the same palette under the second decoder: **4 of 8 series** are left. North and central carry the same lightness step, south and the second tariff tier are the same, east and the third are the same, west and the fourth are the same. Four pairs of series fall to a single series each, and which member of each pair is being read remains unclear. The report thinks it shows eight subsets of the 3,199 readings, while it shows four. This is not a palette flaw, it is a flaw of **structure**. The palette crossed two hue families with four lightness steps and filled eight cells; when the hue family cannot be read, the cross table is left with columns and no rows. If color is the only encoding channel, this result is unavoidable, and increasing the number of colors in the palette does not help. #### Where Redundant Encoding Is Added ```python # redundant.py — the shape channel is added. Where it is added determines the result. SEQUENTIAL = [(p[0], p[1], i % SHAPES) for i, p in enumerate(PALETTE)] BY_COLLISION = [(p[0], p[1], i // SHAPES) for i, p in enumerate(PALETTE)] def shaped(x, y): return reduced_decoder(x, y) or x[2] != y[2] print(f"{'layer':29s}{'full decoder':>13s}{'reduced decoder':>17s}") for name, code, dec in (("color only", PALETTE, reduced_decoder), ("color + shape (sequential)", SEQUENTIAL, shaped), ("color + shape (by collision)", BY_COLLISION, shaped)): full = distinguishable([k[:2] for k in code], full_decoder) print(f"{name:28s} {full:12d} {distinguishable(code, dec):16d}") print(f"the shapes distributed sequentially number {SHAPES} and start over at the " f"{SHAPES}th series; since the color collision is also {SHAPES} series apart, " f"two collisions overlap") ``` ``` layer full decoder reduced decoder color only 8 4 color + shape (sequential) 8 4 color + shape (by collision) 8 8 the shapes distributed sequentially number 4 and start over at the 4th series; since the color collision is also 4 series apart, two collisions overlap ``` The middle line is the most expensive finding of this lesson. The shape channel was added, four separate shapes were distributed to the series in sequence, and the number read by the narrowed decoder **stayed at four**. Redundant encoding was added, but nothing improved. The reason is in the last line. There are four shapes and they start over at the fifth series; the color collision of the palette is also exactly four series apart. When the two cycles overlap, the same colliding pairs this time share both the same step and the same shape. The second channel collides together with the first, in the same place. The third line gives the correction: when the shape is distributed not by order but **by collision** — that is, when the first four series get one shape and the next four series get another — the narrowed decoder also distinguishes all eight. The number of shapes used **fell** from four to two, and the result improved. The measure of redundant encoding is not the number of channels but where the channel changes. #### Legend Box and Transition Count ```python # transitions.py — the legend box sends the reader back and forth between the # display and the box. The right-edge position is a third channel and carries the order. ROWS = 12 lowest, highest = min(m for _, m, _ in SERIES), max(m for _, m, _ in SERIES) position = [round((m - lowest) / (highest - lowest) * (ROWS - 1)) for _, m, _ in SERIES] overlapping = sum(1 for i in range(len(SERIES)) for j in range(i + 1, len(SERIES)) if position[i] == position[j]) unique = [i for i in range(len(SERIES)) if all(position[i] != position[j] for j in range(len(SERIES)) if j != i)] print(f"right-edge positions ({ROWS} rows): {position}") print(f"of {len(SERIES)} series, {len(unique)} have a unique position, " f"{overlapping} pair falls on the same row") box = 2 * len(SERIES) ordered = 2 + 2 * (len(SERIES) - len(unique)) print(f"{'layout':27s}{'transitions':>14s}{'distinguishable series':>23s}") for name, g, s in (("legend box", box, distinguishable(BY_COLLISION, shaped)), ("ordered legend box", ordered, distinguishable(BY_COLLISION, shaped)), ("direct labeling", 0, len(SERIES))): print(f"{name:26s} {g:13d} {s:22d}") ``` ``` right-edge positions (12 rows): [2, 1, 4, 0, 3, 1, 6, 11] of 8 series, 6 have a unique position, 1 pair falls on the same row layout transitions distinguishable series legend box 16 8 ordered legend box 6 8 direct labeling 0 8 ``` Distinguishability is a necessary condition, not a sufficient one. The eight series may be told apart from one another and the reader still does not know which one is which region; this they learn only by looking at the legend box. The measure here is **the number of transitions**: how many times the gaze goes back and forth between the display and the legend box. An ordinary legend box asks for one trip there and one trip back for each series; eight series come to **16 transitions**. When the order in the legend box is made the same as the order of the series on the right edge, the reader solves the whole list with a single mapping and the transitions fall to **6**. The remaining four come from the single pair that falls on the same row on the right edge: at the resolution of 12 rows, south and the second tariff tier cannot be separated, and those two series have to be matched one by one. Direct labeling brings the number of transitions down to **zero**, because the name is written where the series ends and there is no matching job left. Ordering and direct labeling are also layers of redundant encoding; their difference is that they carry not distinguishability but **recognition**. All three layouts distinguish the same eight series, but the work they ask of the reader is 16, 6 and 0. #### The Ceiling of a Single Display ```python # ceiling.py — how many series can be carried in a single display. DISTINCT_LEVELS = len(range(0, LEVELS, SEPARATION)) CEILING = DISTINCT_LEVELS * SHAPES print(f"of {LEVELS} lightness steps, {DISTINCT_LEVELS} are distinguishable at an " f"interval of {SEPARATION}; with {SHAPES} shapes the ceiling is {CEILING} series") print(f"{'series':>6s}{'distinguishable':>18s}{'parts':>7s}") for n in (8, 12, 16, 20, 24, 30, 40): code = [(0, (i % DISTINCT_LEVELS) * SEPARATION, (i // DISTINCT_LEVELS) % SHAPES) for i in range(n)] e = distinguishable(code, shaped) print(f"{n:6d} {e:17d} {max(1, -(-n // CEILING)):6d}") print(f"at most {CEILING} series can be distinguished in a single display; above that " f"the number of series cannot be read back without splitting the display") ``` ``` of 9 lightness steps, 5 are distinguishable at an interval of 2; with 4 shapes the ceiling is 20 series series distinguishable parts 8 8 1 12 12 1 16 16 1 20 20 1 24 20 2 30 20 2 40 20 2 at most 20 series can be distinguished in a single display; above that the number of series cannot be read back without splitting the display ``` The ceiling comes out of two numbers. Of nine lightness steps, five can be distinguished at a resolution of two steps; its product with four shapes is **20**. Up to twenty every series is distinguished, at 24 series the number stays at 20 and four become unreadable; at 40 series it is still 20. This is the hardest limit of the display and it cannot be exceeded by design. If more than twenty series are to be carried, the display is split: the series are divided into groups that do not exceed the ceiling and each group is drawn in its own display. The price is that comparison between groups can no longer be read from the same display. The rule of the topic comes from here — **a display carries as many series as the number of distinguishable series; the rest counts as drawn but not reported**. #### Summary - Color is not a name but two numbers, and in the model it is made of a hue family and a lightness step; the reading resolution is two steps. - The default palette that crosses two hue families with four lightness steps has eight series distinguished by the full decoder and four by the decoder that does not read the hue distinction. - When the shape channel is distributed in sequence, the number stays at four, because the shape starts over after four series and overlaps with the color collision; when distributed by collision it rises to eight with two shapes. - Distinguishability does not give recognition: the same eight series ask for 16 transitions with a legend box, 6 with an ordered legend, and 0 with direct labeling. - The ceiling of a single display is the product of the number of distinguishable steps and the number of shapes; with five steps and four shapes it is 20 series, and above that the display has to be split. #### Next Step Over eight lessons a single display was built correctly. The question type was matched to the channel, the shape of the distribution came down to a number, the slice without a baseline was measured, the point where the relationship becomes unreadable was found, the slope and the season were separated on the time axis, the weight of the area channel was counted, the four distortions were printed one by one, and finally how many series are distinguished was measured. The measure was the same in all of them: a display is as correct as the number the reader can read back. A report, however, is not made of a single display. The reader meets more than one number at a glance, and they are encoded with different channels and stand at different resolutions. The next topic carries the measure here: how many criteria can be read back at a glance, how the same criterion can be moved without moving how it measures, and whether a report gives the same number when it is produced a second time. The reader is no longer a person who reads a number back, but a person who makes a decision with that number. ## Reporting Counting how much of a finding survives when it is carried to the person who will decide: the number of metrics that can be read back from a dashboard and the number of decisions the layout supports, how many ways a metric can be gamed without moving what it measures, the number the refresh lag between the data model and the data source writes into the report, how many steps it takes to build the narrative from finding to recommendation and the gap a skipped step leaves in the reader, and the number of items that must be pinned to produce the same report a second time. ### Dashboard Design Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/dashboard-design The same twelve metrics are encoded in three layouts, and the number of metrics read back at a glance is counted for each: 1 in the layout arranged by insertion order, 2 when every metric is given its own display, and 9 in the layout split by decision. The number of supported decisions is 0, 0 and 3 respectively; when the glance budget is raised to twelve, the first two layouts have 2 and 6 metrics read back but still support no decision. In the length channel, the 1.861 difference between two ratios read back separately with errors of 1.50% and 3.08% is read as 2.500, an error of 34.3%. Across eight lessons, a single display was built correctly: a value was encoded into a channel, the reader read it back, and the error in between was counted. A report is not made of a single display. The person who will make the decision meets several numbers at a glance; every number may be encoded correctly and the page may still support no decision at all. The measure of this lesson is two numbers: **the number of metrics that can be read back at a glance** and **the number of supported decisions**. The dashboard here is not an operations tool. The dashboard lesson of the Observability and Operations course dealt with the on-call engineer's screen and audited it by asking "does the panel answer a question"; alert rules and threshold design were built there and **are not repeated here**. The question here is a different one: can the number be read back from its channel, and does it have a comparison anchor that will turn it into a decision. - **RP1.** The dataset is inherited from the Data Collection and Preparation course and is **fictional**: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions, tariff tiers of 10/25/40 m³. The seed is written into the code. - **RP2.** The report is also **fictional**: the twelve metrics and three decisions come from the period-end report of the unit that runs the measurement network. - **RP3.** A glance is **five fixations**. This is a decision; the same audit is run again with budgets of three, five, eight and twelve. - **RP4.** The reading resolution is written into the decoder: a text bar counts as **whole cells** (20 cells), a color step resolves only to the **midpoint** of the step (5 steps), and a bare number is printed with two decimals. The channel carries the magnitude; the label carries the sign. - **RP5.** The error threshold is **5%**; its rationale is that the smallest comparison in the report is of this magnitude. The scale ceiling is also a decision and is written together with the metric definition. - **RP6.** A metric counts as **read back** if all three conditions are met at once: being within the glance budget, having an error below the threshold, having a comparison anchor. #### Metric, Channel and Anchor All twelve metrics are computed from the fictional dataset. Three of them come from the Statistical Analysis and Exploratory Data Analysis course: the east mean of 24.674 m³, its difference from the north of 4.463 m³, and the 69 subscribers with no readings at all. ```python # report.py — MODEL: the fictional dataset from K03 is generated with the same seed; # a fictional period-end report is built on top of it. Each metric is encoded into a # channel and the reader reads it back. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DISCARDED = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (DISCARDED if r() < 0.046 else SUBSCRIBERS).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {k["subscriber_id"] for k in SUBSCRIBERS} READINGS = {} for k in SUBSCRIBERS + DISCARDED: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in VALID: READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t r() def readings_of(period, region=None): return [READINGS[k["subscriber_id"]][period] for k in SUBSCRIBERS if (region is None or k["region"] == region) and period in READINGS.get(k["subscriber_id"], {})] def missing_rate(period, region=None): group = [k for k in SUBSCRIBERS if region is None or k["region"] == region] return sum(1 for k in group if period not in READINGS.get(k["subscriber_id"], {})) / len(group) d2, d3 = readings_of(2), readings_of(3) ordered = sorted(d3) pooled = {b[0]: sum((readings_of(d, b[0]) for d in (1, 2, 3)), []) for b in REGION} mean = {b: sum(v) / len(v) for b, v in pooled.items()} missing = {b[0]: missing_rate(3, b[0]) for b in REGION} most_missing = max(missing, key=lambda b: missing[b]) # (code, name, value, unit, scale ceiling). The ceiling is a decision; the channel encodes by it. METRICS = [ ("O1", "period 3 total consumption", sum(d3), "m3", 20000.0), ("O2", "mean per subscriber", sum(d3) / len(d3), "m3", 40.0), ("O3", "median consumption", (ordered[418] + ordered[419]) / 2, "m3", 40.0), ("O4", "missing reading rate", missing_rate(3) * 100, "%", 50.0), ("O5", "zero reading count", sum(1 for v in d3 if v == 0.0), "count", 50.0), ("O6", "subscriber share above 40 m3", sum(1 for v in d3 if v > 40) / len(d3) * 100, "%", 5.0), ("O7", "highest region mean", max(mean.values()), "m3", 40.0), ("O8", "east-north mean gap", mean["east"] - mean["north"], "m3", 10.0), ("O9", "period 2-3 change rate", (sum(d3) / len(d3) - sum(d2) / len(d2)) / (sum(d2) / len(d2)) * 100, "%", 20.0), ("O10", "subscriber share in 10-25 tier", sum(1 for v in d3 if 10 <= v < 25) / len(d3) * 100, "%", 100.0), ("O11", "subscribers with no readings", sum(1 for k in SUBSCRIBERS if not READINGS.get(k["subscriber_id"])), "count", 100.0), ("O12", "highest missing reading rate (" + most_missing + ")", missing[most_missing] * 100, "%", 50.0), ] NAME = {o[0]: o[1] for o in METRICS} VALUE = {o[0]: o[2] for o in METRICS} UNIT = {o[0]: o[3] for o in METRICS} CEILING = {o[0]: o[4] for o in METRICS} print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, " f"{sum(len(v) for v in READINGS.values())} readings, {len(REGION)} regions") print(f"report is built on period 3: {len(d3)} readings\n") for k, name, d, unit, ceiling in METRICS: print(f" {k:<4} {name:<38} {d:>10.3f} {unit}") ``` ``` fictional dataset (seed 20260218): 1329 subscribers, 3199 readings, 5 regions report is built on period 3: 838 readings O1 period 3 total consumption 15353.190 m3 O2 mean per subscriber 18.321 m3 O3 median consumption 17.845 m3 O4 missing reading rate 36.945 % O5 zero reading count 28.000 count O6 subscriber share above 40 m3 0.955 % O7 highest region mean 24.674 m3 O8 east-north mean gap 4.463 m3 O9 period 2-3 change rate -6.910 % O10 subscriber share in 10-25 tier 74.105 % O11 subscribers with no readings 69.000 count O12 highest missing reading rate (west) 38.806 % ``` #### Three Channels of the Same Number The rationale for the threshold also comes out of this list: the smallest comparison the report carries is between the dataset-wide missing reading rate and the rate of the highest region. ```python # Each channel is an encoder and decoder pair; the reading resolution is written in the decoder. CELLS, STEPS, THRESHOLD = 20, 5, 0.05 def encode(channel, x, ceiling): if channel == "number": return round(abs(x), 2) if channel == "length": return max(0, min(CELLS, int(abs(x) / ceiling * CELLS + 0.5))) return max(0, min(STEPS - 1, int(abs(x) / ceiling * STEPS))) def decode(channel, code, ceiling): if channel == "number": return code if channel == "length": return code * ceiling / CELLS return (code + 0.5) * ceiling / STEPS def readback_error(channel, k): x, ceiling = VALUE[k], CEILING[k] return abs(decode(channel, encode(channel, x, ceiling), ceiling) - abs(x)) / abs(x) smallest = (VALUE["O12"] - VALUE["O4"]) / VALUE["O4"] print(f"smallest comparison carried by the report: dataset-wide missing reading rate " f"{VALUE['O4']:.3f} versus\n{NAME['O12']} {VALUE['O12']:.3f}, a gap of " f"{smallest * 100:.1f} percent. The read-back error threshold is therefore {THRESHOLD * 100:.0f} percent.\n") print(f"{'metric':<6}{'source value':>16} {'channel':<9}{'encoded':>9}" f"{'read back':>13}{'error %':>9}") for k in ("O9", "O12"): for channel in ("number", "length", "color"): print(f"{k:<6}{VALUE[k]:>16.3f} {channel:<9}{encode(channel, VALUE[k], CEILING[k]):>9}" f"{decode(channel, encode(channel, VALUE[k], CEILING[k]), CEILING[k]):>13.3f}" f"{readback_error(channel, k) * 100:>9.2f}") ``` ``` smallest comparison carried by the report: dataset-wide missing reading rate 36.945 versus highest missing reading rate (west) 38.806, a gap of 5.0 percent. The read-back error threshold is therefore 5 percent. metric source value channel encoded read back error % O9 -6.910 number 6.91 6.910 0.00 O9 -6.910 length 7 7.000 1.31 O9 -6.910 color 1 6.000 13.17 O12 38.806 number 38.81 38.810 0.01 O12 38.806 length 16 40.000 3.08 O12 38.806 color 3 35.000 9.81 ``` The error of the bare number is close to zero, and that does not make it the best channel: whether the 38.81 read without error is good or bad cannot be read from the source itself. The color step, which divides the range into five buckets, exceeds the threshold on both metrics. #### Three Layouts, Two Numbers The first layout keeps the insertion order of the metrics and prints all of them as bare numbers. The second gives each metric a display and an anchor without changing the order. The third divides them into sections by decision. ```python # RP3: a glance is five fixations; the same audit runs below for four budgets. BUDGET = 5 DECISION = [("which region gets an extra reading round", ("O4", "O12", "O8")), ("should the tariff tier boundary be reviewed", ("O3", "O6", "O10")), ("should the period 3 total be refreshed", ("O5", "O9", "O11"))] ORDER = ["O1", "O2", "O9", "O4", "O5", "O3", "O6", "O7", "O8", "O10", "O11", "O12"] KIND = {"O1": "length", "O2": "color", "O9": "color", "O4": "color", "O5": "length", "O3": "length", "O6": "color", "O7": "length", "O8": "number", "O10": "color", "O11": "length", "O12": "color"} LAYOUT = { "insertion order": [[(k, "number", k in ("O8", "O9")) for k in ORDER]], "one display per metric": [[(k, KIND[k], True) for k in ORDER]], "split by decision": [[(k, "number" if k == "O8" else "length", True) for k in kk] for _, kk in DECISION], } def audit(sections, budget=BUDGET): """A metric counts as read back only if three conditions hold at once: being within the glance budget, having an error below the threshold, having an anchor.""" read_back = [k for b in sections for k, channel, anchor in b[:budget] if anchor and readback_error(channel, k) <= THRESHOLD] return read_back, [name for name, kk in DECISION if all(k in read_back for k in kk)] print(f"{len(METRICS)} metrics, {len(DECISION)} decisions, error threshold {THRESHOLD * 100:.0f} percent. " f"Last four columns: metrics read back / supported decisions.\n") print(f"{'layout':<24}{'shown':>11}{'glances':>9}" + "".join(f"{'budget ' + str(b):>10}" for b in (3, 5, 8, 12))) for name, sections in LAYOUT.items(): print(f"{name:<24}{sum(len(b) for b in sections):>11}{len(sections):>9}" + "".join( f"{'/'.join(str(len(x)) for x in audit(sections, b)):>10}" for b in (3, 5, 8, 12))) ``` ``` 12 metrics, 3 decisions, error threshold 5 percent. Last four columns: metrics read back / supported decisions. layout shown glances budget 3 budget 5 budget 8 budget 12 insertion order 12 1 1/0 1/0 1/0 2/0 one display per metric 12 1 1/0 2/0 4/0 6/0 split by decision 9 3 9/3 9/3 9/3 9/3 ``` With the five-fixation budget the three layouts have 1, 2 and 9 metrics read back respectively; the number of supported decisions is 0, 0 and 3. The real information is in the last column. Even when the budget is raised to twelve, that is, when the reader is assumed to scan the page from start to finish, the first two layouts support no decision. What is missing is not space but **co-occurrence**: a decision requires three metrics to be anchored and under the threshold at the same time, and in the first two layouts this triple is not completed at any budget. A page that has six of the twelve metrics read back can support zero decisions. The cost of the third layout is counted too: it shows nine metrics, leaves three out, and asks for **three glances** instead of one. #### The Page the Reader Sees ```python # What the reader sees is printed; "<" marks a metric that is read back. def bar(k, channel): if channel == "length": n = encode("length", VALUE[k], CEILING[k]) return "#" * n + "." * (CELLS - n) if channel == "color": return f"step {encode('color', VALUE[k], CEILING[k]) + 1}/{STEPS}".ljust(CELLS) return f"{VALUE[k]:.2f} {UNIT[k]}".ljust(CELLS) def show(title, sections, names=None, cut=None): print(f"--- {title} " + "-" * (52 - len(title))) for i, b in enumerate(sections): if names: print(f" [decision: {names[i]}]") for j, (k, channel, anchor) in enumerate(b[:cut] if cut else b): mark = " <" if j < BUDGET and anchor and readback_error(channel, k) <= THRESHOLD else "" print(f" {j + 1:>2}. {NAME[k]:<36}{bar(k, channel)}" f"{' has anchor' if anchor else ' no anchor'}{mark}") show("layout 1 — insertion order (five rows per glance)", LAYOUT["insertion order"], cut=BUDGET) print() show("layout 3 — split by decision", LAYOUT["split by decision"], [k[0] for k in DECISION]) actual = VALUE["O12"] - VALUE["O4"] read = decode("length", encode("length", VALUE["O12"], CEILING["O12"]), CEILING["O12"]) - \ decode("length", encode("length", VALUE["O4"], CEILING["O4"]), CEILING["O4"]) print(f"\nO4 error {readback_error('length', 'O4') * 100:.2f} percent, " f"O12 error {readback_error('length', 'O12') * 100:.2f} percent — both below the threshold.") print(f"difference between them: source {actual:.3f}, read back {read:.3f}, " f"error of the difference {abs(read - actual) / actual * 100:.1f} percent") ``` ``` --- layout 1 — insertion order (five rows per glance) --- 1. period 3 total consumption 15353.19 m3 no anchor 2. mean per subscriber 18.32 m3 no anchor 3. period 2-3 change rate -6.91 % has anchor < 4. missing reading rate 36.95 % no anchor 5. zero reading count 28.00 count no anchor --- layout 3 — split by decision ------------------------ [decision: which region gets an extra reading round] 1. missing reading rate ###############..... has anchor < 2. highest missing reading rate (west) ################.... has anchor < 3. east-north mean gap 4.46 m3 has anchor < [decision: should the tariff tier boundary be reviewed] 1. median consumption #########........... has anchor < 2. subscriber share above 40 m3 ####................ has anchor < 3. subscriber share in 10-25 tier ###############..... has anchor < [decision: should the period 3 total be refreshed] 1. zero reading count ###########......... has anchor < 2. period 2-3 change rate #######............. has anchor < 3. subscribers with no readings ##############...... has anchor < O4 error 1.50 percent, O12 error 3.08 percent — both below the threshold. difference between them: source 1.861, read back 2.500, error of the difference 34.3 percent ``` The last two lines show the gap the layout does not close. The two ratios in the first section are read back in the length channel with errors of 1.50% and 3.08% respectively, both well below the threshold. But the work the reader does in that section is not reading the individual values but reading **the difference between them**. The difference, 1.861 points in the source, is read as 2.500: an error of 34.3%. Because the two small errors do not fall in the same direction, they grow in the difference. This shows not that the threshold was chosen wrongly but that the thing being measured has changed. If a decision rests on a difference, the read-back error must be measured **on the difference**: while the smallest comparison in the report is 5%, the 34.3% the channel produces in the difference is large enough to overturn that comparison. #### Summary - A metric counts as read back if all three conditions are met at once: being within the glance budget, having a read-back error below the threshold, having a comparison anchor. - The same twelve metrics are read back 1, 2 and 9 times respectively in three layouts at a glance of five fixations; the number of supported decisions is 0, 0 and 3. When the budget is raised to twelve, the first two layouts have 2 and 6 metrics read back but still support no decision. - The cost of the layout split by decision is counted: it shows nine metrics, leaves three out and asks for three glances instead of one. The bare number is read without error but, having no anchor, does not feed a decision. - The 1.861 difference between two ratios read with errors of 1.50% and 3.08% individually is read as 2.500; the error of the difference is 34.3%. #### Next Step This lesson took the metrics as given and measured only how they stand on the page. The metric itself was not questioned: what the missing reading rate is a rate of, who is in its denominator, at what moment a reading counts as a reading. From the moment a metric starts producing decisions, it becomes possible to game it without moving what it measures. The next lesson counts this: in how many ways the same missing reading rate can be gamed with nothing changing in the measurement network, how much each way moves the metric, and how many of these ways close when the metric is redefined. ### Key Performance Indicators Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/key-performance-indicators The gameability of a metric is counted: the reading coverage rate, which stands at 60.948%, is tried in six ways without a single reading changing in the measurement network, and five of them exceed the gaming threshold — denominator narrowing 64.286, counting zero as a reading 63.055, window widening 81.791, rounding 61.000, target normalization 93.766. A six-item redefinition lowers the number of ways exceeding the threshold from 5 to 0; a way that does not exceed the threshold is not counted as closed, because its movement of 0.037 points comes not from the definition but from the region sizes. When 106 readings are really lost in the measurement network, the tight definition drops from 60.948 to 53.273, while two of the six ways of the loose definition keep the number above its pre-drop value in the same drop. The previous lesson took the twelve metrics as given and measured only their place on the page. One of the metrics, the missing reading rate, fed two of the three decisions. From the moment a number starts producing decisions it gains a new property: the person who reports it can move the number **without touching the thing it measures at all**. The measure of this lesson is **gameability**: in how many ways and by how many points a metric can be moved without moving the thing it measures at all. Metric design itself is not built here. Error budgets, service level indicators and the choice of operational metrics were built in the Observability and Reliability, System Administration, and DevOps Culture and Fundamentals courses; here a single metric is taken and only how loose its definition is gets counted. - **RP7.** The dataset is inherited from the Data Collection and Preparation course and is **fictional**: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code. - **RP8.** The metric is also **fictional**: the **reading coverage rate**, the ratio of the subscribers with a nonzero reading in period 3 to the subscribers registered at the start of the period. - **RP9.** Moving the number **never touches** the measurement network: no reading is added, deleted or changed. Only the definition or the presentation of the metric changes. - **RP10.** Each of the six ways is defensible on its own; none of them fabricates data. - **RP11.** The gaming threshold is **0.05 points**. Its rationale is that a single subscriber in the denominator is worth 0.075 points; the threshold filters out every movement smaller than one subscriber. - **RP12.** The real drop in the measurement network is also **fictional** and is generated with a separate seed. #### Metric ```python # metric.py — MODEL: the fictional dataset from K03 is generated with the same seed; a single # key performance indicator (KPI) is built on top of it. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DISCARDED = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (DISCARDED if r() < 0.046 else SUBSCRIBERS).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {k["subscriber_id"] for k in SUBSCRIBERS} READINGS = {} for k in SUBSCRIBERS + DISCARDED: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in VALID: READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t r() TARGET = 65.0 def coverage(period=3, zero_counts=False, narrow_denominator=False, wide=False, data=None): data = data if data is not None else READINGS denominator = [k for k in SUBSCRIBERS if not narrow_denominator or data.get(k["subscriber_id"])] numerator = 0 for k in denominator: o = data.get(k["subscriber_id"], {}) windows = (2, 3) if wide else (period,) if any(d in o and (zero_counts or o[d] > 0.0) for d in windows): numerator += 1 return numerator / len(denominator) * 100, numerator, len(denominator) baseline, numerator0, denominator0 = coverage() print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, " f"{sum(len(v) for v in READINGS.values())} readings") print(f"metric — reading coverage rate (period 3): {numerator0} / {denominator0} = {baseline:.3f}") ``` ``` fictional dataset (seed 20260218): 1329 subscribers, 3199 readings metric — reading coverage rate (period 3): 810 / 1329 = 60.948 ``` What the metric wants to measure can be said in one sentence: how many of the subscribers the measurement network was able to reach within the period. When this sentence is turned into a number, at least four decisions are made and none of them is written in the sentence. Who is in the denominator, is zero a reading, when does the period close, with how many decimals is the number written. #### Six Ways Each way loosens one of these decisions. None of them fabricates data, none touches the measurement network, and each can be defended in a meeting with a single sentence. ```python # Six ways. None of them touches the measurement network: no reading is added, deleted or changed. # Each way loosens one item of the definition or of the presentation, and each is defensible. THRESHOLD_POINTS = 0.05 TIGHT = {"denominator", "zero", "window", "presentation", "aggregation", "target"} def region_rate(data): rates = [] for b, _, _ in REGION: group = [k for k in SUBSCRIBERS if k["region"] == b] rates.append(sum(1 for k in group if data.get(k["subscriber_id"], {}).get(3, 0.0) > 0.0) / len(group) * 100) return sum(rates) / len(rates) def ways(data, fixed=frozenset()): t = coverage(data=data)[0] w = [("denominator narrowing", "denominator", coverage(data=data, narrow_denominator=True)[0], "%"), ("counting zero as a reading", "zero", coverage(data=data, zero_counts=True)[0], "%"), ("window widening (period 2 or 3)", "window", coverage(data=data, wide=True)[0], "%"), ("rounding to a whole number", "presentation", float(round(t)), "%"), ("unweighted region average", "aggregation", region_rate(data), "%"), ("presenting by target normalization", "target", t / TARGET * 100, "% of target")] return [(name, item, t if item in fixed else v, unit) for name, item, v, unit in w] print(f"baseline metric {baseline:.3f}, target {TARGET:.0f}. One subscriber in the denominator is " f"worth {100 / denominator0:.3f} points, so the gaming threshold is {THRESHOLD_POINTS} points.\n") print(f"{'way':<38}{'reported value':>15}{'movement':>9} unit") for name, item, v, unit in ways(READINGS): print(f"{name:<38}{v:>15.3f}{v - baseline:>9.3f} {unit}") one = coverage(zero_counts=True, narrow_denominator=True, wide=True) print(f"{'first three ways together':<38}{one[0]:>15.3f}{one[0] - baseline:>9.3f} " f"% ({one[1]} / {one[2]})") ``` ``` baseline metric 60.948, target 65. One subscriber in the denominator is worth 0.075 points, so the gaming threshold is 0.05 points. way reported value movement unit denominator narrowing 64.286 3.338 % counting zero as a reading 63.055 2.107 % window widening (period 2 or 3) 81.791 20.843 % rounding to a whole number 61.000 0.052 % unweighted region average 60.985 0.037 % presenting by target normalization 93.766 32.818 % of target first three ways together 87.381 26.433 % (1101 / 1260) ``` Without a single reading changing in the measurement network, the same metric can be reported anywhere between 60.948% and 93.766%. The defensibility of the ways is real too. Removing the 69 subscribers with no readings at all from the denominator looks reasonable, because the meters of these subscribers could never be reached. A zero reading is a reading: the meter was read, no consumption came out. Saying that a subscriber with a reading in period 2 or 3 was "reached" can also be defended. The two largest movements come from the two ways that look most innocent. Window widening carries 20.843 points, target normalization 32.818 points. The second changes the **unit** of the number: the 93.766 that is written is not a coverage rate but the percentage of the target that was realized. If the unit is not written under the same line, the reader reads it back as the coverage itself. The ways are not mutually exclusive. When the first three are applied together, the number written into the report, staying in the same unit, becomes 87.381: 1,101 subscribers out of 1,260. Three defensible decisions made by three different people in three different meetings carry the metric 26.433 points up without anyone fabricating anything. The hardest to notice of the ways is the first, and the reason lies in the concept of the previous lesson. If the report line says only "64.286%", the reader has nothing with which to tell apart a ratio with a numerator of 810 and a denominator of 1,260 from a ratio with a numerator of 810 and a denominator of 1,329. Whether a ratio can be read back depends not on an anchor but on **its numerator and denominator standing next to it**; a ratio whose two parts are not written hides its own definition. #### Redefinition The redefinition is six items and each item binds one way. Then the same six ways are tried again. ```python # The redefinition is six items and each item binds one way: the denominator is the subscriber # registered at the start of the period, zero is not counted as a reading, the window closes at # period 3, the rate is written with three decimals, aggregation is subscriber-weighted, and # target normalization is forbidden. def try_ways(fixed, data=READINGS): t = coverage(data=data)[0] return [(name, abs(v - t) > THRESHOLD_POINTS, v) for name, item, v, unit in ways(data, fixed)] loose, tight = try_ways(frozenset()), try_ways(TIGHT) print(f"{'way':<38}{'loose definition':>18}{'tight definition':>18}") for (name, la, lv), (_, ta, tv) in zip(loose, tight): print(f"{name:<38}{('moved ' + format(lv, '.3f')) if la else 'did not move':>18}" f"{('moved ' + format(tv, '.3f')) if ta else 'did not move':>18}") unclosed = [a for a in loose if not a[1]] print(f"\nways exceeding the threshold: {sum(1 for a in loose if a[1])} under the loose definition, " f"{sum(1 for a in tight if a[1])} under the tight definition ({len(loose)} ways tried)") for name, _, v in unclosed: print(f"not closed, did not work on this dataset: {name} — {abs(v - baseline):.3f} points, " f"smaller than the {100 / denominator0:.3f} points one subscriber is worth") ``` ``` way loose definition tight definition denominator narrowing moved 64.286 did not move counting zero as a reading moved 63.055 did not move window widening (period 2 or 3) moved 81.791 did not move rounding to a whole number moved 61.000 did not move unweighted region average did not move did not move presenting by target normalization moved 93.766 did not move ways exceeding the threshold: 5 under the loose definition, 0 under the tight definition (6 ways tried) not closed, did not work on this dataset: unweighted region average — 0.037 points, smaller than the 0.075 points one subscriber is worth ``` The number of ways exceeding the threshold falls from 5 to 0. The real warning in the table is in the fifth row. The unweighted region average did not exceed the threshold even under the loose definition, because in this dataset the coverage rates of the five regions are close enough to each other. Counting this way as "closed" would mean accepting that what keeps the number in place is not the definition but the size distribution of the regions. If one of the regions shrank or its coverage diverged, the same way would exceed the threshold, and if the definition has no counterpart for it, nothing would stop it. This also determines how the gameability number is to be read: **a way that was tried and did not move is not a closed way.** What is counted is how many ways the definition binds, not how many ways worked in this run. Writing all six items into a definition lengthens the metric, and this is a cost. When the one-line expression "reading coverage rate" turns into a six-item definition, the reader of the report can no longer read the metric back at a glance. The way the cost is paid is to write the definition not into the report but into the place the report rests on: the report line carries only the numerator, the denominator and the definition identifier. #### Real Drop The tight definition may have a cost: if making the metric ungameable also makes it insensitive to the thing it measures, there is no gain. To measure this, a real loss is produced in the measurement network. ```python # Real drop: a portion of the period 3 readings is really lost in the measurement network. # This is not moving the number; the thing that is measured itself changes. DROP_DATA, lost = {}, 0 for k in SUBSCRIBERS: o = dict(READINGS.get(k["subscriber_id"], {})) if 3 in o and rng(SEED + 91000 + k["subscriber_id"])() < 0.14: del o[3] lost += 1 DROP_DATA[k["subscriber_id"]] = o new = coverage(data=DROP_DATA)[0] print(f"period 3 readings lost in the measurement network: {lost}") print(f"number given by the tight definition: {baseline:.3f} -> {new:.3f} ({new - baseline:+.3f} points)\n") print(f"{'loose definition ways, same drop':<38}{'reported value':>15}" f"{'before the drop':>17} unit") masking = 0 for name, item, v, unit in ways(DROP_DATA): masking += 1 if v >= baseline else 0 print(f"{name:<38}{v:>15.3f}{('above' if v >= baseline else 'below'):>17} {unit}") print(f"\n{masking} of the six ways keep the number above its pre-drop value in the same real drop;\n" f"one of them does it by changing the unit") ``` ``` period 3 readings lost in the measurement network: 106 number given by the tight definition: 60.948 -> 53.273 (-7.675 points) loose definition ways, same drop reported value before the drop unit denominator narrowing 56.190 below % counting zero as a reading 55.079 below % window widening (period 2 or 3) 81.490 above % rounding to a whole number 53.000 below % unweighted region average 53.470 below % presenting by target normalization 81.959 above % of target 2 of the six ways keep the number above its pre-drop value in the same real drop; one of them does it by changing the unit ``` The tight definition passes the loss through as it is: when 106 readings are lost, the number drops from 60.948 to 53.273, that is by 7.675 points. Being ungameable does not mean being insensitive. Two of the six ways of the loose definition keep the number above its pre-drop value in the same real drop. Window widening writes 81.490, because most owners of the readings lost in period 3 still have their period 2 reading. Target normalization writes 81.959. Both reports come from the same dataset, both are computed correctly, and one says coverage fell while the other says it rose. The gameability of the metric finds its full meaning here. Moving the number does not just make a good number look better; **it makes bad news invisible.** A real loss of seven points can pass without being reported at all in a metric with a loose definition. There is one more distinction to note. Window widening covering up the drop is not a sign of bad faith: a subscriber whose period 2 reading stands has really been reached, and the person who chooses that way does not have to be doing it knowingly. When the definition is not written, the way can be chosen without anyone preferring it; because it is not written with which window the metric was computed in the previous period, the use of a different window this period goes unnoticed. Gameability is not a personality trait but a property of the definition. #### Summary - Gameability is a countable quantity: in how many ways and by how many points a metric can be moved without touching the thing it measures at all. - When the coverage rate, which stands at 60.948%, is tried in six ways, five exceed the threshold of 0.05 points and the number written into the report can stand anywhere between 60.948 and 93.766. - The two largest movements are window widening (20.843 points) and target normalization (32.818 points); the second changes the unit of the number, and if the unit is not written the reader reads it back as coverage. - The six-item redefinition lowers the ways exceeding the threshold from 5 to 0; a way that does not exceed the threshold is not counted as closed, because its movement of 0.037 points comes not from the definition but from the size distribution of the regions. - The ways are not mutually exclusive: when the first three are applied together the number, staying in the same unit, becomes 87.381 (1,101 subscribers out of 1,260) and is carried 26.433 points up. - When 106 readings are really lost in the measurement network, the tight definition drops by 7.675 points; two of the six ways of the loose definition keep the number above its pre-drop value in the same drop. #### Next Step This lesson fixed the definition of the metric, but fixing the definition is not fixing the number. The tight definition says "the window closes at period 3"; when a reading that belongs to the closed window enters the system after the report was produced, the same definition applied to the same dataset gives a different number. The report stands on a source, and that source is connected to the report either by a live query, or by a periodic extract, or by a pre-aggregated model. The next lesson runs the three connection schemes with the same metric, counts the difference among them and shows which questions the report data model closes forever. ### Business Intelligence Tools Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/business-intelligence-tools The same metric gives a different value under three refresh schemes and the difference is counted: in a report produced twelve days after the close of period 3, the reading rate comes out 56.659 with a live query, 48.081 with a periodic extract and 36.042 with a pre-aggregated model; the final value at the source is 63.055. The same lag moves mean consumption by only 0.038 m³. The grain and the field set that the report data model fixes permanently close three of six questions: the model that stores only the mean gives a wrong answer to one question by 0.192 m³, while the model that stores the sum and the count answers three questions correctly. Three schemes and five report days produce fifteen reports, five distinct values and a spread of 27.013 points. The previous lesson fixed the definition of the metric with six items. One of the items said "the window closes at period 3". Meter readings are not ready the moment the window closes: some enter the source on the same day, some weeks later. When the same definition is applied to the same dataset on two different days, it gives two different numbers, and both conform to the definition. The report stands on top of the source, and there is a tool in between. This tool is referred to not by its product but by its **parts**, and it has four of them: the **data source connector** that connects to the source, the **report data model** built on top of the source, the **refresh scheme** that determines how often the model is refreshed, and the presentation layer that puts the number in front of the reader. The measure of this lesson comes from the second and the third parts: **the number that the refresh lag writes into the report** and **the number of questions the model closes permanently**. - **RP13.** The dataset is inherited from the Data Collection and Preparation course and is **fictional**: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code. - **RP14.** The **arrival lag** of each reading is also fictional and is drawn with a separate seed. The lag does not change the value of the reading, it only determines at which moment the reading becomes visible. - **RP15.** Two metrics are tracked: the period 3 **reading rate** (subscribers with a period 3 reading / registered subscribers) and the period 3 **mean consumption**. - **RP16.** Three refresh schemes: the **live query** reads the source at report time, the **periodic extract** is refreshed every seven days, the **pre-aggregated model** is aggregated once at period close and is not aggregated again. - **RP17.** The report data model is at the **region-period grain**; two designs are tried and they differ only in the fields they store. - **RP18.** The accuracy comparison is not made with `==`; the threshold is $10^{-9}$ and its rationale is the digit accumulation of floating-point summation. #### Connecting to the Source ```python # refresh.py — MODEL: the fictional dataset from K03 is generated with the same seed; a # fictional arrival lag is added to each reading and the report is read under three refresh schemes. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DISCARDED = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (DISCARDED if r() < 0.046 else SUBSCRIBERS).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {k["subscriber_id"] for k in SUBSCRIBERS} READINGS = {} for k in SUBSCRIBERS + DISCARDED: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in VALID: READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t r() # Each reading enters the source some time after the period closes. The lag is fictional and is # drawn with a separate seed; it does not change the value of the reading, only its visibility. ARRIVAL_LAG = {} for k in SUBSCRIBERS: r = rng(SEED + 120000 + k["subscriber_id"]) for d in sorted(READINGS.get(k["subscriber_id"], {})): ARRIVAL_LAG[(k["subscriber_id"], d)] = pick(r(), [0.58, 0.19, 0.13, 0.07, 0.03]) * 6 def snapshot(day): """The source snapshot in which the readings whose lag does not exceed the day value are visible.""" g = {} for (a, d), lag in ARRIVAL_LAG.items(): if lag <= day: g.setdefault(a, {})[d] = READINGS[a][d] return g def metrics(data): """Two metrics: the period 3 reading rate and the period 3 mean consumption.""" d3 = [o[3] for o in data.values() if 3 in o] return len(d3) / len(SUBSCRIBERS) * 100, sum(d3) / len(d3), len(d3) FULL = snapshot(10 ** 6) final = metrics(FULL) print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(ARRIVAL_LAG)} readings") print(f"arrival lag distribution (days): " + ", ".join( f"{g}: {sum(1 for v in ARRIVAL_LAG.values() if v == g)}" for g in (0, 6, 12, 18, 24))) print(f"final figures at the source — reading rate {final[0]:.3f}, " f"period 3 mean {final[1]:.3f} m3 ({final[2]} readings)") ``` ``` fictional dataset (seed 20260218): 1329 subscribers, 3199 readings arrival lag distribution (days): 0: 1841, 6: 608, 12: 444, 18: 213, 24: 93 final figures at the source — reading rate 63.055, period 3 mean 18.321 m3 (838 readings) ``` Of the 3,199 readings, 1,841 are in the source on the closing day and the remaining 1,358 enter on later days. The final figures of the source are also read from this list, and the rest of the lesson is measured against these two numbers. #### Three Refresh Schemes ```python # Three refresh schemes. The report is produced 12 days after the close of period 3; the schemes # differ only in how far behind they see the source. REPORT_DAY = 12 SCHEME = [("live query", REPORT_DAY, 0), ("periodic extract (every 7 days)", 7, REPORT_DAY - 7), ("pre-aggregated model", 0, REPORT_DAY)] print(f"report day: {REPORT_DAY} days after the close of period 3\n") print(f"{'scheme':<32}{'refresh lag':>19}{'reading rate':>13}{'mean':>10}" f"{'readings':>9}{'gap from final':>15}") for name, cut, lag in SCHEME: k, o, n = metrics(snapshot(cut)) print(f"{name:<32}{str(lag) + ' days':>19}{k:>13.3f}{o:>10.3f}{n:>9}" f"{o - final[1]:>+10.3f} m3") print(f"{'final value at source':<32}{'-':>19}{final[0]:>13.3f}{final[1]:>10.3f}{final[2]:>9}" f"{0.0:>+10.3f} m3") ``` ``` report day: 12 days after the close of period 3 scheme refresh lag reading rate mean readings gap from final live query 0 days 56.659 18.128 753 -0.193 m3 periodic extract (every 7 days) 5 days 48.081 18.160 639 -0.161 m3 pre-aggregated model 12 days 36.042 18.166 479 -0.155 m3 final value at source - 63.055 18.321 838 +0.000 m3 ``` The choice among the three schemes is not a choice of accuracy but a trade-off. The live query gives the freshest number and loads its cost onto the source every time a report is opened; the pre-aggregated model never strains the source and pays its cost in freshness; the periodic extract sits between the two. Which end of the trade-off was chosen is not written into the number itself. Same day, same source, same metric definition: the reading rate comes out 56.659, 48.081 and 36.042. All three are computed correctly and all three are below the final 63.055. That even the live query is incomplete shows that the lag is not in the report but in the source itself: on the report day those readings have not entered yet. The second column carries the real lesson. The same lag moves mean consumption only between 18.128 and 18.166, that is by only 0.038 m³; it moves the reading rate by 20.617 points. The reason is that the two are a **count** and a **ratio**: late-arriving readings enlarge the numerator, not the denominator, but their values are not systematically different from those of the early arrivals. Of two metrics standing side by side on a dashboard, one can be almost insensitive to the refresh scheme while the other depends on it entirely, and this comes not from the definition of the metric but from its **kind**. #### What the Report Data Model Fixes The report data model is not a copy of the source; it is a structure whose **grain** and **field set** have been chosen. Neither can be loosened afterwards: once the model is aggregated at the region-period grain, every question at the subscriber level is closed permanently. ```python # Report data model: a structure built on top of the source, with its grain and fields fixed. # Two designs are tried; both are at the region-period grain, their fields differ. MODEL_A = {} # only the mean is stored MODEL_B = {} # the sum and the count are stored for b, _, _ in REGION: v = [FULL[k["subscriber_id"]][3] for k in SUBSCRIBERS if k["region"] == b and 3 in FULL.get(k["subscriber_id"], {})] MODEL_A[b] = sum(v) / len(v) MODEL_B[b] = (sum(v), len(v)) raw3 = sorted(o[3] for o in FULL.values() if 3 in o) single = [k["subscriber_id"] for k in SUBSCRIBERS if FULL.get(k["subscriber_id"], {}).get(3, 0.0) > 0][0] CORRECT = {"dataset mean": sum(raw3) / len(raw3), "regional mean (east)": MODEL_A["east"], "reading rate": len(raw3) / len(SUBSCRIBERS) * 100, "median consumption": (raw3[len(raw3) // 2 - 1] + raw3[len(raw3) // 2]) / 2, "subscribers above 40 m3": float(sum(1 for v in raw3 if v > 40)), "one subscriber reading": FULL[single][3]} ANSWER = { "dataset mean": (sum(MODEL_A.values()) / len(MODEL_A), sum(x[0] for x in MODEL_B.values()) / sum(x[1] for x in MODEL_B.values())), "regional mean (east)": (MODEL_A["east"], MODEL_B["east"][0] / MODEL_B["east"][1]), "reading rate": (None, sum(x[1] for x in MODEL_B.values()) / len(SUBSCRIBERS) * 100), "median consumption": (None, None), "subscribers above 40 m3": (None, None), "one subscriber reading": (None, None)} EPS = 1e-9 def status(v, d): if v is None: return "unproducible" return "correct" if abs(v - d) < EPS else f"wrong ({v - d:+.3f})" print(f"{'question':<26}{'source (subscriber-period grain)':>34}{'model A':>18}{'model B':>18}") counter = [0, 0, 0] for s, d in CORRECT.items(): a, b = ANSWER[s] for i, v in enumerate((d, a, b)): counter[i] += 1 if v is not None and abs(v - d) < EPS else 0 print(f"{s:<26}{format(d, '.3f'):>34}{status(a, d):>18}{status(b, d):>18}") print(f"\ncorrectly answered questions: source {counter[0]}/6, model A {counter[1]}/6, " f"model B {counter[2]}/6") ``` ``` question source (subscriber-period grain) model A model B dataset mean 18.321 wrong (-0.192) correct regional mean (east) 22.394 correct correct reading rate 63.055 unproducible correct median consumption 17.845 unproducible unproducible subscribers above 40 m3 8.000 unproducible unproducible one subscriber reading 21.570 unproducible unproducible correctly answered questions: source 6/6, model A 1/6, model B 3/6 ``` The two models are at the same grain and their only difference is the field they store: one stores the mean, the other the sum and the count. This difference opens and closes three questions. When the sum and the count are stored, the dataset mean is built correctly by adding up the sums of the regions and dividing by their counts; when only the mean is stored, it cannot be built, because the weights have been thrown away. The first row is the most important result of this lesson. Model A does not say **unproducible** to the dataset mean question; it gives a number that is wrong by 0.192 m³. The mean of the five regional means is mathematically the result of an operation and appears on a dashboard as the dataset mean. An answer that cannot be produced makes itself known; an answer that is produced wrong does not. The bottom three rows are the cost of the grain. The median, the number of subscribers above a threshold and the reading of a single subscriber cannot be produced by either model, because these are not additive fields: it is impossible to build the median of a region from the medians of its parts, or the dataset median from the medians of five regions. The report data model looks like a speed decision; **it is a decision that narrows the set of questions that can be asked**, and it has closed three of the six questions permanently. #### The Same Metric, Fifteen Reports ```python # The same metric, the same source, five different report days. The pre-aggregated model is # aggregated once at period close and is not aggregated again; the extract is refreshed every seven days. DAYS = (0, 6, 12, 24, 40) CUT = {"live query": lambda g: g, "periodic extract": lambda g: 7 * (g // 7), "pre-aggregated model": lambda g: 0} print(f"{'scheme':<26}" + "".join(f"{'day ' + str(g):>10}" for g in DAYS)) values = set() for name, f in CUT.items(): row = [] for g in DAYS: v = metrics(snapshot(f(g)))[0] values.add(round(v, 3)) row.append(v) print(f"{name:<26}" + "".join(f"{v:>10.3f}" for v in row)) print(f"{'final value at source':<26}" + "".join(f"{final[0]:>10.3f}" for g in DAYS)) print(f"\n{len(CUT)} schemes x {len(DAYS)} report days = {len(CUT) * len(DAYS)} reports, " f"{len(values)} distinct values; all are computed correctly") print(f"widest spread: {max(values) - min(values):.3f} points " f"({min(values):.3f} and {max(values):.3f})") print(f"the number written into the report therefore cannot be written alone: " f"the value, the data cut-off and the refresh lag are written together") ``` ``` scheme day 0 day 6 day 12 day 24 day 40 live query 36.042 48.081 56.659 63.055 63.055 periodic extract 36.042 36.042 48.081 60.873 63.055 pre-aggregated model 36.042 36.042 36.042 36.042 36.042 final value at source 63.055 63.055 63.055 63.055 63.055 3 schemes x 5 report days = 15 reports, 5 distinct values; all are computed correctly widest spread: 27.013 points (36.042 and 63.055) the number written into the report therefore cannot be written alone: the value, the data cut-off and the refresh lag are written together ``` The first two rows converge: the live query settles on the final value on the twenty-fourth day, the periodic extract on the fortieth. The third row never converges and stays at 36.042, because it was aggregated once at period close and never again. This row is not a flaw but a decision; the flaw is that the decision is not written in the report. The fifteen reports have five distinct values, and there are 27.013 points between the two at the extremes. All of them were produced from the same source, with the same definition and with correct computation. The reader of a report cannot resolve the difference between "reading rate 36.042" and "reading rate 63.055" by looking at the number; the only things that resolve the difference are the **data cut-off** and the **refresh lag** standing next to the number. If these two are not in the report, the value itself cannot be read back. #### Summary - A business intelligence tool is referred to by four parts: data source connector, report data model, refresh scheme and presentation layer. The measure comes from the second and the third parts. - In a report produced on the same day, the reading rate comes out 56.659 with a live query, 48.081 with a periodic extract and 36.042 with a pre-aggregated model; the final value at the source is 63.055 and all three numbers are correct. - The same lag moves mean consumption by only 0.038 m³ and the reading rate by 20.617 points; the sensitivity comes not from the definition of the metric but from its kind. - The report data model fixes a grain and a field set: the model that stores only the mean produces the dataset mean wrong by 0.192 m³, while the model that stores the sum and the count answers the same question correctly. - The three non-additive questions (the median, the number of subscribers above a threshold, the reading of a single subscriber) cannot be produced by either model; the grain decision closes three of six questions permanently. - Three schemes and five report days produce fifteen reports, five distinct values and a spread of 27.013 points; the only things that resolve the difference are the data cut-off and the refresh lag standing next to the number. #### Next Step The number now comes with the right definition, from the right model and with a written cut-off. What the reader holds, however, is still a number. For a decision to emerge, the number has to sit in a chain: finding, comparison, candidate cause, uncertainty and recommendation. When a link of the chain is missing, the reader completes it on their own, and which direction they take while completing it is not written down. The next lesson builds this chain and, by removing the links one at a time, counts the inference the reader is forced to make; exactly where the margin of variation of the Statistical Analysis and Exploratory Data Analysis course stands in this chain is also settled there. ### Building Narrative with Data Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/building-narrative-with-data When one step is removed from the five-step chain that runs from finding to recommendation, the premises the reader is forced to complete on their own are counted: removing the finding leaves 4 inferences open, removing the comparison 3, removing the candidate cause 2, removing the uncertainty 3, removing the recommendation 3. Of the eleven wrong defaults the chain carries, nine strengthen the recommendation and two weaken it — the gaps are not filled at random. The place of the margin of variation of the Statistical Analysis and Exploratory Data Analysis course is settled: if the margin is never written, 4 inferences and 3 wrong defaults remain; if it is written only at the end in a limitations paragraph, 1 inference and 1 wrong default remain; when it is written next to the finding and in the uncertainty step, both go to zero. The previous three lessons fixed the number: the layout was split to support the decision, the definition of the metric was made ungameable, and the data cut-off and the refresh lag were written next to the number. What the reader holds is still a number. For a number to turn into a decision it has to sit in a chain, and if the links of the chain are missing, the reader completes them on their own. The measure of this lesson is **the gap that a skipped step leaves**: the number of inferences the reader is forced to make when a link is removed. A second number is written next to it: how many of the defaults of these inferences are wrong with respect to the dataset, and how many of those wrong defaults **strengthen** the recommendation. - **RP19.** The dataset is inherited from the Data Collection and Preparation course and is **fictional**: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code. - **RP20.** The finding comes from the Statistical Analysis and Exploratory Data Analysis course: the east mean is 24.674 m³ (557 readings), the north 20.210 m³ (921 readings), the gap 4.463 m³. **The test computation is not repeated here**; the numbers are read from there. - **RP21.** Both numbers of the margin are read from the Statistical Analysis and Exploratory Data Analysis course: when the region labels were reshuffled 2,000 times, the observed gap was not exceeded in any run; the spread of the four defensible decisions on mass, sample, zero readings and the number of periods is 0.206 m³. - **RP22.** The narrative chain is **fictional**: five steps, sixteen premises, fifteen of them needed to defend the recommendation. - **RP23.** When a step is removed, the reader is assumed to complete the premises of that step on their own. The default of each premise and the direction of the default are written in the lesson. - **RP24.** The recommendation is also **fictional**: a review of the tariff tier boundary in the east region. #### The Chain ```python # narrative.py — MODEL: the fictional dataset from K03 is generated with the same seed; on top # of it a five-step narrative chain that runs from finding to recommendation is built. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DISCARDED = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (DISCARDED if r() < 0.046 else SUBSCRIBERS).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {k["subscriber_id"] for k in SUBSCRIBERS} READINGS = {} for k in SUBSCRIBERS + DISCARDED: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in VALID: READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t r() MEANS = {} for b, _, _ in REGION: v = [t for k in SUBSCRIBERS if k["region"] == b for t in READINGS.get(k["subscriber_id"], {}).values()] MEANS[b] = (sum(v) / len(v), len(v)) GAP = MEANS["east"][0] - MEANS["north"][0] WIDEST = max(MEANS.values())[0] - min(MEANS.values())[0] ORDERED = sorted(MEANS, key=lambda b: -MEANS[b][0]) # Narrative chain: each step carries a set of premises. Premise fields: code, step, # content, the default the reader falls to if the step is missing, whether the default is wrong # for the dataset, whether it is needed to defend the recommendation, whether the default # strengthens the recommendation (+1/-1). PREMISES = [ ("O1", "finding", f"east mean {MEANS['east'][0]:.3f} m3", "number unknown", False, True, 0), ("O2", "finding", f"north mean {MEANS['north'][0]:.3f} m3", "number unknown", False, True, 0), ("O3", "finding", f"numbers rest on {MEANS['east'][1]} and {MEANS['north'][1]} readings", "observation count is large", True, True, 1), ("O4", "finding", "the number in the finding was written with its margin", "number is exact", True, True, 1), ("O5", "comparison", f"gap {GAP:.3f} m3", "gap unknown", False, True, 0), ("O6", "comparison", f"gap is {GAP / MEANS['north'][0] * 100:.1f} percent of north", "gap is small", True, True, -1), ("O7", "comparison", f"widest gap of the five regions {WIDEST:.3f} m3 ({ORDERED[0]}-{ORDERED[-1]})", "gap is the only important gap", True, True, 1), ("O8", "candidate cause", "candidate: east base consumption is higher", "no cause exists", False, True, 0), ("O9", "candidate cause", "rival candidate: reading coverage gap", "a single cause exists", True, True, 1), ("O10", "candidate cause", "rival candidate: tariff tier distribution", "a single cause exists", True, False, 1), ("O11", "uncertainty", "K05: not exceeded in 2,000 reshuffles", "margin not measured", False, True, 0), ("O12", "uncertainty", "K05: spread of four decisions 0.206 m3", "margin is zero", True, True, 1), ("O13", "uncertainty", "what the margin does not cover was named", "margin covers everything", True, True, 1), ("O14", "recommendation", "recommended action was written", "action is the reader's choice", True, True, -1), ("O15", "recommendation", "reversibility of the action was written", "action is reversible", True, True, 1), ("O16", "recommendation", "which metric the action will be measured by", "no measurement will be done", True, True, 1), ] STEPS = ["finding", "comparison", "candidate cause", "uncertainty", "recommendation"] print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, " f"{sum(len(v) for v in READINGS.values())} readings") print(f"finding: east {MEANS['east'][0]:.3f} m3 ({MEANS['east'][1]} readings), " f"north {MEANS['north'][0]:.3f} m3 ({MEANS['north'][1]} readings), gap {GAP:.3f} m3") print(f"chain: {len(STEPS)} steps, {len(PREMISES)} premises, " f"{sum(1 for o in PREMISES if o[5])} of them needed to defend the recommendation") for a in STEPS: print(f" {a:<15} " + ", ".join(o[0] for o in PREMISES if o[1] == a)) ``` ``` fictional dataset (seed 20260218): 1329 subscribers, 3199 readings finding: east 24.674 m3 (557 readings), north 20.210 m3 (921 readings), gap 4.463 m3 chain: 5 steps, 16 premises, 15 of them needed to defend the recommendation finding O1, O2, O3, O4 comparison O5, O6, O7 candidate cause O8, O9, O10 uncertainty O11, O12, O13 recommendation O14, O15, O16 ``` Here the narrative is not a style but a **set of premises**. If the recommendation is to be defensible, the premises standing behind it must have been given to the reader; every premise that is not given is produced by the reader. What the model does is count this production. The name of the third step is also deliberately not "cause" but **candidate cause**. The report writes that the base consumption of the east region is higher as a candidate, puts the two rival candidates beside it, and does not decide among them; there is no mechanism in the dataset that would make this decision. Writing the candidates is not an evasion; it is the only way to keep the reader from falling onto a single candidate. #### The Skipped Step ```python # One step is removed from the chain and the premises the reader is forced to complete on # their own are counted as needed. Wrong default: a completion that is not true for the dataset. def inference_gap(removed=None, dropped=frozenset()): missing = [o for o in PREMISES if o[5] and (o[1] == removed or o[0] in dropped)] wrong = [o for o in missing if o[4]] return missing, wrong, sum(1 for o in wrong if o[6] > 0) print(f"{'removed step':<22}{'reader inference':>20}{'wrong default':>19}" f"{'strengthening':>21}") for a in [None] + STEPS: e, y, g = inference_gap(a) print(f"{a or 'none (full chain)':<22}{len(e):>20}{len(y):>19}{g:>21}") print(f"\nall wrong defaults in the whole chain and their directions") for o in PREMISES: if o[4]: print(f" {o[0]:<4} {o[1]:<16} the reader falls to this default: {o[3]:<30}" f"{'strengthens the recommendation' if o[6] > 0 else 'weakens the recommendation'}") every_wrong = [o for o in PREMISES if o[4]] print(f"of {len(every_wrong)} wrong defaults in total, {sum(1 for o in every_wrong if o[6] > 0)} " f"strengthen the recommendation, {sum(1 for o in every_wrong if o[6] < 0)} weaken it") ``` ``` removed step reader inference wrong default strengthening none (full chain) 0 0 0 finding 4 2 2 comparison 3 2 1 candidate cause 2 1 1 uncertainty 3 2 2 recommendation 3 3 2 all wrong defaults in the whole chain and their directions O3 finding the reader falls to this default: observation count is large strengthens the recommendation O4 finding the reader falls to this default: number is exact strengthens the recommendation O6 comparison the reader falls to this default: gap is small weakens the recommendation O7 comparison the reader falls to this default: gap is the only important gap strengthens the recommendation O9 candidate cause the reader falls to this default: a single cause exists strengthens the recommendation O10 candidate cause the reader falls to this default: a single cause exists strengthens the recommendation O12 uncertainty the reader falls to this default: margin is zero strengthens the recommendation O13 uncertainty the reader falls to this default: margin covers everything strengthens the recommendation O14 recommendation the reader falls to this default: action is the reader's choice weakens the recommendation O15 recommendation the reader falls to this default: action is reversible strengthens the recommendation O16 recommendation the reader falls to this default: no measurement will be done strengthens the recommendation of 11 wrong defaults in total, 9 strengthen the recommendation, 2 weaken it ``` The first row of the table confirms the definition of the chain: in the full chain no inference is left to the reader, because all fifteen of the fifteen premises needed to defend the recommendation have been written. This row is not a measure of success but the measuring instrument's own zero point; the remaining five rows are read against it. None of the five steps is free. When the finding is removed, four inferences are left to the reader; when the uncertainty or the recommendation is removed, three each; when the comparison is removed, three; when the candidate cause is removed, two. The numbers are close to each other and, read alone, do not say which step is more expensive to skip. The second and the third columns make the distinction. Of the eleven wrong defaults, nine **strengthen** the recommendation and only two weaken it. The gaps are not filled at random: when the reader cannot see a margin next to a number, the reader counts the margin as zero; when the reader cannot see a rival candidate cause, the reader assumes there is a single cause; when the number of observations is not written, the reader accepts that the number comes from a large dataset. All three make the recommendation look sturdier than it is. The result is a methodological rule. Skipping a narrative step makes the recommendation **easier to defend**, and this ease is independent of the author's intent. The two weakening defaults do not break this rule; on the contrary, they explain it: when the reader does not see the size of the gap, the reader counts it as small, and when the recommended action is not written, the reader chooses the action. These two are not skipped, because they do not serve the author; the steps that are skipped are the steps that serve once they are skipped. #### The Place of the Margin At this point the place of the margin of variation of the Statistical Analysis and Exploratory Data Analysis course in the chain can be settled. Three placements are tried. ```python # The place of the margin in the narrative is tried with three placements. The margin was # measured in K05 and is not recomputed here; what is asked is at which step of the chain it stands. PLACEMENT = [("margin not written at all", {"O4", "O11", "O12", "O13"}), ("margin only at the end, in a limitations paragraph", {"O4"}), ("margin next to the finding and in the uncertainty step", set())] print(f"{'placement of the margin':<58}{'inference':>9}{'wrong':>8}{'strengthening':>15}") for name, dropped in PLACEMENT: e, y, g = inference_gap(dropped=dropped) print(f"{name:<58}{len(e):>9}{len(y):>8}{g:>15}") print("\n--- chain of the third placement ---------------------------------") for a in STEPS: print(f" [{a}]") for o in PREMISES: if o[1] == a: print(f" {o[0]:<4} {o[2]}{'' if o[5] else ' (not needed)'}") ``` ``` placement of the margin inference wrong strengthening margin not written at all 4 3 3 margin only at the end, in a limitations paragraph 1 1 1 margin next to the finding and in the uncertainty step 0 0 0 --- chain of the third placement --------------------------------- [finding] O1 east mean 24.674 m3 O2 north mean 20.210 m3 O3 numbers rest on 557 and 921 readings O4 the number in the finding was written with its margin [comparison] O5 gap 4.463 m3 O6 gap is 22.1 percent of north O7 widest gap of the five regions 10.896 m3 (east-west) [candidate cause] O8 candidate: east base consumption is higher O9 rival candidate: reading coverage gap O10 rival candidate: tariff tier distribution (not needed) [uncertainty] O11 K05: not exceeded in 2,000 reshuffles O12 K05: spread of four decisions 0.206 m3 O13 what the margin does not cover was named [recommendation] O14 recommended action was written O15 reversibility of the action was written O16 which metric the action will be measured by ``` The second placement is the common one, and it leaves one inference open. The margin is written, but it is written at the end: while reading the finding the reader saw no interval next to the number and had already formed the "the number is exact" default. The limitations paragraph that comes at the end does not take this default back; it only clears the author's record. The margin has not one place but **two places**, and the first is next to the number in the finding. The second is the uncertainty step, and there the margin says two different things. The gap that was not exceeded in the 2,000 reshuffles of the Statistical Analysis and Exploratory Data Analysis course closes a single question: would this gap come out if the region labels were distributed at random. The 0.206 m³ spread of the four defensible decisions is an entirely different uncertainty, and the reshuffling margin does not measure it at all. This is why a third premise stands in the chain: **what the margin does not cover is named.** When it is not named, the default the reader falls to is "the margin covers everything", and this default too is among those that strengthen the recommendation. The last three premises of the chain tie the recommendation to an action: the action is written, whether it is reversible is written, the metric by which it will be measured is written. The last one ties the previous two lessons to this point. If the metric is not written, the recommendation is never tested; if the metric is written, its definition has to be tight, otherwise the number that evaluates the recommendation can be moved without moving what it measures. #### Summary - The narrative is not a style but a five-step chain of premises: finding, comparison, candidate cause, uncertainty, recommendation. The chain carries sixteen premises and fifteen of them are needed to defend the recommendation. - When a step is removed, the number of inferences left to the reader is between 2 and 4; this number alone does not say which step is more expensive. - Of the chain's eleven wrong defaults, nine strengthen the recommendation and two weaken it; skipping a step makes the recommendation easier to defend, and this ease is independent of the author's intent. - The margin of variation has not one place but two: next to the number in the finding and in the uncertainty step. A separate limitations paragraph at the end leaves 1 inference open, and that inference is the "the number is exact" default. - The uncertainty step also names what the margin does not cover: the 2,000-reshuffle margin of the Statistical Analysis and Exploratory Data Analysis course closes only the question of random distribution of the labels, and does not measure the 0.206 m³ spread of the four decisions. - The third step writes not the cause but the candidate causes; the last three premises of the chain tie the recommendation to an action, to the reversibility of the action and to the metric that will test it. #### Next Step The report now stands on a chain: the numbers are in a layout that can be read back, the definitions of the metrics are tight, the cut-off of the source is written, the links of the narrative are complete. All of this concerns producing a single report **once**. Whether the same report can be produced again three months later, by another person, so that it gives the same numbers, was never asked. The next lesson counts this: how many items have to be fixed to produce the same report a second time, how much the number moves when one is missing, and how many items the report adds on top of the items counted by the Data Collection and Preparation course and the Data Analysis with Python course. ### Report Reproducibility Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/report-reproducibility The items that must be fixed to produce the same report a second time are counted: the report's own 11 items are added on top of the 39 decisions in the preparation ledger of the Data Collection and Preparation course and the 9 environment items of the Data Analysis with Python course, for a total of 59. Of the eleven items, 10 move the report when released alone — the reading rate goes anywhere between 36.042 and 100.000; the one item that does not move it, released together with another item, carries 87.168 to 94.432. The data cut-off and the refresh scheme change all seven lines of the seven-line report, while the presentation items change only one; the eleven items move 26 lines in total. The previous lesson sat the report on a chain of premises and counted the links of the chain. That report was produced once. Whether another person could reproduce the same report three months later so that it gives the same numbers was never asked. A report is not a file but **the product of a run**, and there is a decision at every step of the run. Part of this question was closed earlier. The Data Collection and Preparation course wrote the preparation pipeline as seventeen steps and thirty-nine decisions; when the run ledger was kept, thirty-nine of the thirty-nine decisions became readable after the run. The Data Analysis with Python course went one step further and wrote the working environment as a list of nine items: with the lock file, eleven of the seventeen steps gave the same result on two machines, and the remaining six depended on five items that were not written in the lock file at all. **Neither computation is repeated here**; they are taken as input. What this lesson adds is **the report itself**: even when the pipeline is the same and the environment is the same, the report may not be the same. - **RP25.** The dataset is inherited from the Data Collection and Preparation course and is **fictional**: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code. - **RP26.** The **arrival lags** of the readings are also fictional and are generated with the seed of the previous lesson. - **RP27.** The report's items are **fictional**: eleven items, each with a fixed value and an equally defensible alternative. - **RP28.** Thresholds: **0.05 points** for the rate (a single subscriber in the denominator is worth 0.075 points), **0.0005 m³** for the mean — the second is the threshold used by the Data Analysis with Python course. Floating-point comparison is not made with `==`. - **RP29.** The 39 decisions of the Data Collection and Preparation course and the 9 environment items of the Data Analysis with Python course are taken as **input**. - **RP30.** The report is **seven lines**: two headline numbers and five region lines. #### The Report's Items ```python # reproduce.py — MODEL: the fictional dataset from K03 is generated with the same seed; on top # of it the fictional arrival lags and the item list that produces the report are built. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DISCARDED = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (DISCARDED if r() < 0.046 else SUBSCRIBERS).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) VALID = {k["subscriber_id"] for k in SUBSCRIBERS} READINGS = {} for k in SUBSCRIBERS + DISCARDED: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in VALID: READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t r() # Each reading enters the source some time after the period closes. The lag is fictional and is # drawn with a separate seed; it does not change the value of the reading, only its visibility. ARRIVAL_LAG = {} for k in SUBSCRIBERS: r = rng(SEED + 120000 + k["subscriber_id"]) for d in sorted(READINGS.get(k["subscriber_id"], {})): ARRIVAL_LAG[(k["subscriber_id"], d)] = pick(r(), [0.58, 0.19, 0.13, 0.07, 0.03]) * 6 def snapshot(day): """The source snapshot in which the readings whose lag does not exceed the day value are visible.""" g = {} for (a, d), lag in ARRIVAL_LAG.items(): if lag <= day: g.setdefault(a, {})[d] = READINGS[a][d] return g # The report's own items. Each item has a fixed value and an equally defensible alternative; # if the alternative is chosen, the report does not give the same report. ITEMS = [("data cut-off", "12 days", "40 days"), ("refresh scheme", "live query", "pre-aggregated model"), ("denominator definition", "registered subscriber", "subscriber with readings"), ("zero reading", "counted as a reading", "not counted as a reading"), ("observation window", "period 3", "period 2 or 3"), ("subscriber filter", "all subscribers", "all three periods complete"), ("region filter", "five regions", "excluding central"), ("aggregation weighting", "subscriber-weighted", "region-unweighted"), ("presentation decimals", "three decimals", "whole number"), ("presentation unit", "rate", "percent of target"), ("target value", "65", "60")] FIXED = {a: b for a, b, _ in ITEMS} def report(k): """The report that produces the two headline numbers from the item list. Every decision is read from an item.""" cut = 12 if k["data cut-off"] == "12 days" else 40 cut = 0 if k["refresh scheme"] != "live query" else cut data = snapshot(cut) chosen = [a for a in SUBSCRIBERS if k["region filter"] == "five regions" or a["region"] != "central"] if k["subscriber filter"] != "all subscribers": chosen = [a for a in chosen if len(data.get(a["subscriber_id"], {})) == 3] windows = (2, 3) if k["observation window"] != "period 3" else (3,) numerator, values, by_region = [], [], {} for a in chosen: o = data.get(a["subscriber_id"], {}) v = [o[d] for d in windows if d in o and (k["zero reading"] == "counted as a reading" or o[d] > 0)] by_region.setdefault(a["region"], [0, 0]) if v: numerator.append(a) values.extend(v) by_region[a["region"]][0] += 1 by_region[a["region"]][1] += 1 denominator = len(chosen) if k["denominator definition"] == "registered subscriber" else \ len([a for a in chosen if data.get(a["subscriber_id"])]) rate = (sum(x[0] / x[1] for x in by_region.values()) / len(by_region) * 100 if k["aggregation weighting"] != "subscriber-weighted" else len(numerator) / denominator * 100) if k["presentation unit"] != "rate": rate = rate / float(k["target value"]) * 100 decimals = 3 if k["presentation decimals"] == "three decimals" else 0 return round(rate, decimals), round(sum(values) / len(values), 3) THRESHOLD_RATE, THRESHOLD_M3 = 0.05, 0.0005 base_rate, base_mean = report(FIXED) print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(ARRIVAL_LAG)} readings") print(f"report items: {len(ITEMS)}") print(f"report with fixed items: reading rate {base_rate:.3f}, " f"period 3 mean {base_mean:.3f} m3") print(f"thresholds: rate {THRESHOLD_RATE} points, mean {THRESHOLD_M3} m3") ``` ``` fictional dataset (seed 20260218): 1329 subscribers, 3199 readings report items: 11 report with fixed items: reading rate 56.659, period 3 mean 18.128 m3 thresholds: rate 0.05 points, mean 0.0005 m3 ``` None of the eleven items is hidden in a line of code; all of them stand in a list that the report function reads. This is the first condition of fixing: if an item is buried in the code, it cannot even be asked whether it is released or fixed. #### When One Item Is Released ```python # Each item is released one at a time: the alternative value is chosen, the other ten items # stay fixed and the report is produced again. def moves(o, m): return abs(o - base_rate) > THRESHOLD_RATE or abs(m - base_mean) > THRESHOLD_M3 print(f"{'released item':<24}{'alternative':<30}{'reading rate':>12}" f"{'mean':>10}{'moved?':>12}") moved = 0 for name, fixed, alt in ITEMS: k = dict(FIXED) k[name] = alt o, m = report(k) moved += 1 if moves(o, m) else 0 print(f"{name:<24}{alt:<30}{o:>12.3f}{m:>10.3f}" f"{('yes' if moves(o, m) else 'no'):>12}") print(f"\n{moved} of {len(ITEMS)} items move the report when released alone") pair = dict(FIXED) pair["presentation unit"], pair["target value"] = "percent of target", "60" o, m = report(pair) print(f"'target value', which does not move alone, when released together with " f"'presentation unit': {o:.3f}") ``` ``` released item alternative reading rate mean moved? data cut-off 40 days 63.055 18.321 yes refresh scheme pre-aggregated model 36.042 18.166 yes denominator definition subscriber with readings 60.531 18.128 yes zero reading not counted as a reading 54.628 18.803 yes observation window period 2 or 3 80.512 19.020 yes subscriber filter all three periods complete 100.000 18.160 yes region filter excluding central 55.842 17.476 yes aggregation weighting region-unweighted 56.746 18.128 yes presentation decimals whole number 57.000 18.128 yes presentation unit percent of target 87.168 18.128 yes target value 60 56.659 18.128 no 10 of 11 items move the report when released alone 'target value', which does not move alone, when released together with 'presentation unit': 94.432 ``` When a single item is released, the reading rate goes anywhere between 36.042 and 100.000. The other ten items are fixed; the only thing that moves is the one that was released. The sixth row also deserves to be read. The "subscribers who have all three periods" filter is a reasonable sentence, and it makes the metric exactly 100.000, because the filter filters the denominator as well: a subscriber without a period 3 reading does not stay in the dataset anyway. The metric no longer measures anything, and the report says so nowhere. **When a filter touches the denominator of a metric, it can stop the metric from being a metric.** The last row determines how the decision to fix is to be made. When the target value is released alone, the report does not budge, because as long as the presentation unit is "rate" the target enters no computation. Released together with the presentation unit, the same item carries 87.168 to 94.432. **An item that does not move the report in this run is not an item that does not need to be fixed.** Items move the report not one by one but together, and an item looking ineffective may depend on another item being fixed. #### The Second Production ```python # The report is seven lines: two headline numbers and five region lines. The second production # runs with the same item list; then each item is released one at a time and the number of # lines that change is counted. def lines(k): o, m = report(k) s = [f"reading rate {o:.3f}", f"period 3 mean {m:.3f} m3"] cut = 12 if k["data cut-off"] == "12 days" else 40 cut = 0 if k["refresh scheme"] != "live query" else cut data = snapshot(cut) for b, _, _ in REGION: v = [data[a["subscriber_id"]][3] for a in SUBSCRIBERS if a["region"] == b and 3 in data.get(a["subscriber_id"], {})] s.append(f"{b} mean {sum(v) / len(v):.3f} m3 ({len(v)} readings)") return s first = lines(FIXED) print(f"second production with the same item list: " f"{sum(1 for a, b in zip(first, lines(dict(FIXED))) if a == b)}/{len(first)} lines identical") print(f"\n{'released item':<24}{'changed lines':>14} which ones") total = 0 for name, fixed, alt in ITEMS: k = dict(FIXED) k[name] = alt changed = [i for i, (a, b) in enumerate(zip(first, lines(k))) if a != b] total += len(changed) print(f"{name:<24}{len(changed):>14} {', '.join(str(i + 1) for i in changed) or '-'}") print(f"\nwhen each of the {len(ITEMS)} items is released alone, {total} lines change in total " f"(in a {len(first)}-line report)") LEDGER = [("Data Collection and Preparation", 39, "decisions in the 17 steps of the pipeline"), ("Data Analysis with Python", 9, "items of the run environment"), ("this lesson — report", len(ITEMS), "the report's own items")] print(f"\n{'source':<32}{'items':>7} what it fixes") for a, n, s in LEDGER: print(f"{a:<32}{n:>7} {s}") print(f"{'total':<32}{sum(n for _, n, _ in LEDGER):>7}") ``` ``` second production with the same item list: 7/7 lines identical released item changed lines which ones data cut-off 7 1, 2, 3, 4, 5, 6, 7 refresh scheme 7 1, 2, 3, 4, 5, 6, 7 denominator definition 1 1 zero reading 2 1, 2 observation window 2 1, 2 subscriber filter 2 1, 2 region filter 2 1, 2 aggregation weighting 1 1 presentation decimals 1 1 presentation unit 1 1 target value 0 - when each of the 11 items is released alone, 26 lines change in total (in a 7-line report) source items what it fixes Data Collection and Preparation 39 decisions in the 17 steps of the pipeline Data Analysis with Python 9 items of the run environment this lesson — report 11 the report's own items total 59 ``` The second production gives the same result in all seven of the seven lines. This shows that carrying the item list is sufficient; what is fixed is not the code itself but the list the code reads. The items are not of equal weight. The data cut-off and the refresh scheme change all seven lines at once, because both determine which snapshot of the source will be read, and that snapshot lies under every line of the report. The presentation items change only one line. The difference gives a priority list: if a single item is to be carried alongside the report, that item is the data cut-off. The total of fifty-nine items comes from three different places and the three do not substitute for one another. The thirty-nine decisions of the preparation ledger fix the steps of the pipeline but do not know the cut-off of the report. The nine items of the working environment fix the run but do not know the filter of the report. The eleven items of the report guarantee nothing unless both of the others are fixed: two reports running with the same item list still diverge if the pipeline or the environment underneath has changed. #### Summary - Reproducing a report requires fixing 59 items from three sources: the 39 decisions of the preparation ledger, the 9 items of the working environment and the report's own 11 items. - Of the eleven items, 10 move the report when released alone; the reading rate goes anywhere between 36.042 and 100.000. - An item that does not move the report alone is not an item that does not need to be fixed: the target value, released together with the presentation unit, carries 87.168 to 94.432. - When a filter touches the denominator of a metric, it can stop the metric from being a metric: the "subscribers who have all three periods" filter makes the reading rate 100.000 by definition. - Items are not of equal weight: the data cut-off and the refresh scheme change all seven lines, the presentation items change only one; the eleven items move 26 lines in total. - When the item list is carried, the second production is identical in all seven of the seven lines; what is fixed is not the code but the list the code reads. #### Next Step In five lessons the topic built a report from start to finish and wrote down the decision at every step. When the same twelve metrics were encoded in three layouts, 1, 2 and 9 metrics were read back at a glance and the number of supported decisions came out 0, 0 and 3. A 60.948% metric was tried in six ways without a single reading changing in the measurement network, and five of them exceeded the threshold. The same metric gave 56.659, 48.081 and 36.042 under three refresh schemes. Nine of the eleven wrong defaults of the chain that runs from finding to recommendation strengthened the recommendation. And producing the report a second time required 59 items. All of this work was done with a silent assumption: the dataset fits on a single machine. The 3,199 rows were read from start to finish in every block, every aggregation was done in a single loop, and trying a filter took seconds. When the measurement network grows, this assumption falls, and the moment it falls it is not the report itself but **the computation itself** that has to be moved. The next topic asks this: when the dataset does not fit on a single machine, which part of the report can be split, how many records does the split part carry across which shuffles, and what does the serial fraction cost. ## Analysis at Scale Counting which part of the same report can be divided once it is built on a cluster that does not fit on a single machine: reducing volume, velocity, and variety to measurable units, the number of passes and the records moved in the map–reduce model, the coordination overhead item that data frame processing on a cluster adds, the values a columnar file layout makes you read and the gain from skipping row groups, the ceiling the serial fraction puts on speedup, and the job size at which distribution is unnecessary. ### Big Data Concept Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/big-data-concept Reducing volume, velocity, and variety to countable units instead of qualitative claims: the 3,199 rows of the measurement network take 914,929 bytes in memory, 286.0 bytes per row, and the threshold at which the dataset no longer fits on a single machine ranges from 7,508,560 to 240,273,951 rows depending on the working space and intermediate share decisions. The same 1,260 meters produce 1,260 rows/period under periodic reading and 3,628,800 rows/period under fifteen-minute reading; the periods left to the threshold fall from 17,875 to 7, and from 168 to 6 in a network that grows 4% per period. Of the 18 fields in five source schemas, 9 cannot be joined and two mapping tables bring that number down to 6; key absence closes, unit and time format conflicts do not. The report was built correctly. The metrics were arranged so that they can be read back at a glance, they were tested for whether they can be gamed, the narrative carried the steps from finding to recommendation without gaps, and the items that must be pinned down so that the report can be produced a second time were counted. Underneath all of this work sits a single unwritten assumption: the dataset fits on one machine. The source is opened, the rows are taken into memory, the computation is done, and the result is written. This topic removes that assumption. The first question is the crudest one: when is a dataset considered "too big for a single machine"? The answer is not an adjective but a number, and without writing down the decisions that produce that number, the number does not exist. The lesson builds volume, velocity, and variety not as three separate qualitative claims but as **six countable units**: bytes and rows, rows/period and latency, the number of distinct schemas and the number of non-joinable fields. - **AS1.** The dataset is inherited from the Data Collection and Preparation, Data Analysis with Python, and Statistical Analysis and Exploratory Data Analysis courses and **is fictional**: 3,199 rows, 1,329 subscribers, three periods, five regions, tariff tiers of 10/25/40 m³. It is generated with the same seed and the output is the same from run to run. **It is a model.** - **AS2.** Bytes per row are measured with a single representation. Which decision determines the bytes that the field types set apart was measured in the Scale and Performance topic of the Data Analysis with Python course; here that decision is an **input** and is taken as a single number. - **AS3.** The working space of a single machine is an assumption and is tested in the table with three values. The intermediate share is the second decision: whether temporary space as large as the dataset itself is requested next to the dataset. - **AS4.** Velocity is two numbers: **rows/period** and **latency**. Latency is the largest number of reading intervals a reading waits in a report that is refreshed periodically. - **AS5.** Variety is two numbers: **the number of distinct schemas** and **the number of non-joinable fields**. A field does not join if it is in a schema with no common key, if it is free text with no mapping, if it is a quantity whose unit must be converted, or if its time format does not fit the dataset's period axis. - **AS6.** No distributed processing engine, file format, or vendor name is written; tools are mentioned by their kind. #### Volume: Bytes and Rows ```python # dataset.py — the FICTIONAL dataset of the regional measurement network. MODEL: instead of # a real dataset a generator that produces the same structure was written; the seed is visible and the # output is the same from run to run. 3199 rows, 1329 subscribers, three periods, five regions. import math, sys SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("center", 0.18, 23)] FIELDS = ("subscriber_id", "region", "period", "household", "satisfaction", "consumption_m3") def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATASET = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfied = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: # meter that failed to register continue sub_id, q = 10001 + i, rng(SEED + 7000 + 10001 + i) for d in range(pick(q(), [0.05, 0.12, 0.21, 0.62])): v = 0.0 if q() < 0.038 else math.floor( b[2] * math.exp((q() + q() + q() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 DATASET.append((sub_id, b[0], d + 1, household, satisfied, v)) q() METERS = len({s[0] for s in DATASET}) total_bytes = sys.getsizeof(DATASET) + sum( sys.getsizeof(s) + sum(sys.getsizeof(a) for a in s) for s in DATASET) BYTES_PER_ROW = total_bytes / len(DATASET) print(f"seed {SEED}; {len(DATASET)} rows, {len(FIELDS)} fields, {METERS} reading meters") print(f"in memory {total_bytes} bytes, {BYTES_PER_ROW:.1f} bytes per row") GIB = 1024 ** 3 print(f"{'working space':<16}{'intermediate share':>20}{'left for dataset':>18}" f"{'threshold rows':>16}{'times':>8}") for gib in (4, 12, 64): for share, label in ((1, "none"), (2, "same as dataset")): left = gib * GIB / share threshold = int(left / BYTES_PER_ROW) print(f"{str(gib) + ' GiB':<16}{label:>20}{gib // share:>14} GiB" f"{threshold:>16}{threshold // len(DATASET):>8}") ``` ``` seed 20260218; 3199 rows, 6 fields, 1260 reading meters in memory 914929 bytes, 286.0 bytes per row working space intermediate share left for dataset threshold rows times 4 GiB none 4 GiB 15017121 4694 4 GiB same as dataset 2 GiB 7508560 2347 12 GiB none 12 GiB 45051365 14082 12 GiB same as dataset 6 GiB 22525682 7041 64 GiB none 64 GiB 240273951 75109 64 GiB same as dataset 32 GiB 120136975 37554 ``` Volume is not a single number but the quotient of two numbers. The working space stands on the numerator side, bytes per row on the denominator side, and the quotient gives a **row count**. The fictional dataset takes 914,929 bytes, 286.0 bytes per row. This number is itself the result of a type decision, and that decision was measured elsewhere; what does not change here is that the threshold can be written as a number. The six rows of the table are the product of two defensible decisions. The working space takes three values, the intermediate share two. The narrowest decision reaches the threshold at 7,508,560 rows, the widest at 240,273,951 rows. The ratio between them is thirty-two times and both ends are defensible: a computation that asks for no intermediate share keeps only running totals, while a computation that asks for temporary space as large as the dataset itself sorts or merges. **The sentence "this dataset does not fit on a single machine" cannot be formed without writing which of the six rows was chosen.** The fictional dataset itself comes close to the threshold in no row: even under the narrowest decision it is one 2,347th of the threshold. The subject of analysis at scale is not this dataset but the speed at which this dataset walks toward that threshold. The next two sections take the middle row as the threshold, 12 GiB of working space with an intermediate share the size of the dataset, and fix 22,525,682 rows. This is a choice and it is written down because it is a choice; when the threshold changes, all the later numbers shift by the same ratio and the direction of the shift can be read from the table. #### Velocity: Rows and Periods ```python # velocity.py — velocity is two numbers: rows/period and latency. Reading frequency is a # decision; the meter itself does not change, yet the number of rows it produces does. THRESHOLD = int((12 * GIB / 2) / BYTES_PER_ROW) # 12 GiB working space, intermediate share the size of the dataset GROWTH = 1.04 # meter growth per period def period_count(rate, growth): total, n, h = len(DATASET), 0, rate while total < THRESHOLD and n < 10 ** 6: total, n, h = total + h, n + 1, h * growth return n print(f"threshold {THRESHOLD} rows; network {METERS} meters, growth per period " f"{round((GROWTH - 1) * 100)} percent") print(f"{'reading frequency':<20}{'reads/period':>14}{'rows/period':>13}" f"{'fixed network':>15}{'growing network':>17}{'latency':>9}") for label, freq in (("periodic", 1), ("daily", 30), ("hourly", 720), ("fifteen-minute", 2880)): rate = METERS * freq print(f"{label:<20}{freq:>14}{rate:>13}{period_count(rate, 1.0):>15}" f"{period_count(rate, GROWTH):>17}{freq:>9}") ``` ``` threshold 22525682 rows; network 1260 meters, growth per period 4 percent reading frequency reads/period rows/period fixed network growing network latency periodic 1 1260 17875 168 1 daily 30 37800 596 82 30 hourly 720 907200 25 18 720 fifteen-minute 2880 3628800 7 6 2880 ``` Velocity is not the growth of the network. The four rows of the table hold **the same 1,260 meters**; the only thing that changes is how many times each meter is read. Under periodic reading the network produces 1,260 rows/period and the threshold is 17,875 periods away; under fifteen-minute reading the same network produces 3,628,800 rows/period and the threshold arrives 7 periods later. Reading frequency looks like a hardware fact, yet it is a decision, and the effect of this decision on the periods left to the threshold is 2,554 times. The fifth column adds the second decision. When the network grows 4% per period, the 17,875 periods under periodic reading drop to 168 periods. A growth of 4% is invisible when looked at period by period; looked at cumulatively, it brings the threshold one hundred times closer. Velocity has two sources and the two are multiplied: frequency and growth. The last column is the second unit of velocity. If a report is refreshed once per period, it takes a reading at most one period to enter the report; but when this wait is written in terms of **reading intervals**, the number changes. Under periodic reading the wait is 1 reading interval, under fifteen-minute reading it is 2,880 reading intervals, and the rows waiting outside the report in the meantime rise from 1,260 to 3,628,800. Raising the frequency does not freshen the data; it enlarges the amount of data that has not been refreshed. #### Variety: Schema and Field ```python # variety.py — variety is two numbers: the number of distinct schemas and the number of # non-joinable fields. MODEL: the schemas of five sources were written as field name, unit, and # time format; the sources themselves were not generated. COMMON = "subscriber_id" SCHEMA = { "subscriber record": [("subscriber_id", "-", "-"), ("region", "name", "-"), ("household", "persons", "-"), ("satisfaction", "1-5", "-")], "meter reading": [("subscriber_id", "-", "-"), ("period", "-", "period no"), ("consumption_m3", "m3", "-")], "field visit": [("meter_serial", "-", "-"), ("value_read", "liter", "-"), ("region_code", "number", "-"), ("visit_day", "-", "day")], "support request": [("request_id", "-", "-"), ("subscriber_id", "-", "-"), ("opened", "-", "timestamp"), ("topic", "text", "-")], "tariff table": [("tier_text", "text", "-"), ("unit_price", "currency", "-"), ("valid_range", "-", "date range")], } def status(schema, unit, time_format): if COMMON not in [a for a, _, _ in SCHEMA[schema]]: return "no key" if unit == "liter": return "unit conversion" if unit == "text": return "unmapped text" if time_format not in ("-", "period no"): return "time format" return "joins" def count(title): reasons, broken = {}, 0 print(f"{title:<18}{'fields':>8}{'common key':>15}{'non-joinable':>14}" f"{'reason':>18}") for name, fields in SCHEMA.items(): d = [status(name, u, t) for _, u, t in fields] n = sum(1 for x in d if x != "joins") for x in d: if x != "joins": reasons[x] = reasons.get(x, 0) + 1 broken += n print(f"{name:<18}{len(fields):>8}" f"{('yes' if COMMON in [a for a, _, _ in fields] else 'no'):>15}" f"{n:>14}{(sorted(set(x for x in d if x != 'joins')) or ['-'])[0]:>18}") print(f"distinct schemas {len(SCHEMA)}, total fields " f"{sum(len(v) for v in SCHEMA.values())}, non-joinable fields {broken}") print("reason breakdown:", ", ".join(f"{k} {v}" for k, v in sorted(reasons.items()))) return broken k1 = count("schema (before)") SCHEMA["field visit"].append(("subscriber_id", "-", "-")) # meter_serial mapping table SCHEMA["tariff table"].append(("subscriber_id", "-", "-")) # tier mapping table print() k2 = count("schema (after)") print(f"two mapping tables reduce non-joinable fields from {k1} to {k2}; " f"key absence closes, unit and time format do not close") ``` ``` schema (before) fields common key non-joinable reason subscriber record 4 yes 0 - meter reading 3 yes 0 - field visit 4 no 4 no key support request 4 yes 2 time format tariff table 3 no 3 no key distinct schemas 5, total fields 18, non-joinable fields 9 reason breakdown: no key 7, time format 1, unmapped text 1 schema (after) fields common key non-joinable reason subscriber record 4 yes 0 - meter reading 3 yes 0 - field visit 5 yes 2 time format support request 4 yes 2 time format tariff table 4 yes 2 time format distinct schemas 5, total fields 20, non-joinable fields 6 reason breakdown: time format 3, unit conversion 1, unmapped text 2 two mapping tables reduce non-joinable fields from 9 to 6; key absence closes, unit and time format do not close ``` The number of distinct schemas does not measure variety. Five sources carry five schemas and this number stays five in both runs; the number that moves is **the number of non-joinable fields**: 9 of the 18 fields cannot be tied to the dataset. The reason breakdown splits these nine into three — 7 fields come from two schemas that have no common key, 1 field is free text with no mapping, and 1 field carries a time format that does not fit the dataset's period axis. The second run applies a decision: a mapping table that converts the meter serial number to the subscriber number is built, along with a second mapping table that converts the tariff tier text to the dataset's tier names. Non-joinable fields fall from 9 to 6, but the reason breakdown shifts: key absence closes completely, while time format conflicts rise from 1 to 3, because the time fields of the two schemas that can now be tied are counted as well. A mapping ties a field; it does not align its unit and its time axis. The direction of the number is also worth noting. In the second run the total number of fields rises from 18 to 20, because the mapping adds one field to each of two schemas; the share of non-joinable fields falls from 50% to 30%, but the absolute number falls by only three. The cost of closing variety is a mapping table for every field that closes, one that must keep being maintained, and these tables do not appear in the dataset itself. When the three units are set side by side, the scale table of the dataset emerges. The volume threshold is 22,525,682 rows, the velocity is 3,628,800 rows/period under fifteen-minute reading, and the variety is 5 schemas and 6 non-joinable fields. None of these three numbers is "big" or "small"; each is the quotient of a pair of decisions, and the decisions are written inside the table. #### Summary - Volume is the quotient of two numbers: the fictional dataset takes 286.0 bytes per row, and the threshold at which it no longer fits on a single machine moves thirty-two times, between 7,508,560 and 240,273,951 rows, depending on the working space and intermediate share decisions. - Velocity is the reading frequency, not the growth of the network: the same 1,260 meters produce 1,260 rows/period under periodic reading and 3,628,800 rows/period under fifteen-minute reading, and the periods left to the threshold fall from 17,875 to 7. - A network growth of 4% per period brings the threshold closer under periodic reading from 17,875 periods to 168 periods; frequency and growth are multiplied. - Latency is the second unit of velocity: in a report refreshed periodically the wait is 1 reading interval under periodic reading and 2,880 reading intervals under fifteen-minute reading, and the waiting rows rise to 3,628,800. - The number of distinct schemas does not measure variety; five schemas are five in both runs, the number that moves is the 9 non-joinable fields out of 18, and two mapping tables bring this number down to 6. - A mapping closes key absence but does not close the unit and time format conflict: the number of fields that do not join because of the time format rises from 1 to 3. #### Next Step This lesson wrote the point at which the dataset no longer fits on a single machine as a row count, but it did not ask what the computation becomes after that point. When the dataset crosses the threshold, the work has to be divided, and the first question of dividing is this: into which pieces can an aggregation be split, and how many records are moved between the pieces. The next lesson writes and runs the simplest abstraction of distributed computation, the map and reduce functions, within the lesson; the measure is **the number of records moved in the shuffle** — how many records move when the same aggregation first reduces and then shuffles and when it does the reverse, and which computation does not enter this model at all. ### Map–Reduce Model Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/mapreduce-model The simplest abstraction of distributed computation is built and run with three functions, and the measure becomes the number of records moved in the shuffle: on the fictional dataset spread across eight source parts, a regional average moves 3,199 records in the order that shuffles first and then reduces and 40 records in the order that reduces locally first and then shuffles, a gain of 79.97 times. The gain depends on the number of keys and falls to 2.53 times on the subscriber key. The same question gives two values with two keys: the east average is 24.674 m³ weighted by rows and 24.814 m³ as the average of subscriber averages, and the records moved by the two passes rise from 40 to 1,305. If the reduce function is not associative the computation does not enter the model: the sum and the maximum do not change across seven different part counts, the median of part medians gives seven different values, and its spread is 0.2175 m³. The previous lesson wrote the point at which the dataset no longer fits on a single machine as a row count: 22,525,682 rows under the chosen decision. Past that point the computation has to be divided, but "divided" is not a recipe by itself. What is divided, what travels between the parts, and how many records travel. This lesson builds and runs the simplest abstraction of distributed computation. The abstraction is three functions: **map** turns a row into a key and an intermediate value, the **shuffle** moves the intermediate values to the reducers by key, and **reduce** closes the intermediate values of the same key into a single value. The measure of the lesson is in the middle step: how many records are moved in the shuffle. The second number next to it is how much of the reduction was done before moving the record. - **AS7.** The dataset is inherited from the Data Collection and Preparation, Data Analysis with Python, and Statistical Analysis and Exploratory Data Analysis courses and **is fictional**: 3,199 rows, 1,329 subscribers, three periods, five regions, tariff tiers of 10/25/40 m³. The seed is 20260218 and the output is the same from run to run. **It is a model.** - **AS8.** No distributed processing engine is installed; the source is taken to be **spread across eight parts** and the three functions are written with the standard library. **It is a model.** - **AS9.** A record moved in the shuffle is a (key, intermediate value) pair sent to the reducers. Raw milliseconds are not measured. - **AS10.** The intermediate value is a decision. For the average the intermediate value is not a single number but a **(total, count)** pair; this choice makes the reduce function associative. - **AS11.** Whether a computation enters the model depends on the reduce function being **associative and commutative** over partial results. The test is done by running: the same computation is run with seven different part counts and the number of distinct numbers that come out is counted. - **AS12.** The part boundary is drawn by row count; the parts are in the order the source gives. #### Three Functions ```python # map_reduce.py — the FICTIONAL dataset of the regional measurement network and three functions: # map, shuffle, reduce. MODEL: no distributed engine is installed; the source is taken to be # spread across PARTS parts and the records moved in the shuffle are counted. import math SEED, RAW, M32, PARTS = 20260218, 1400, 0xFFFFFFFF, 8 REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("center", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tier(v): return "0-10" if v <= 10 else "10-25" if v <= 25 else "25-40" if v <= 40 else "over 40" DATASET = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfied = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: # meter that failed to register continue sub_id, q = 10001 + i, rng(SEED + 7000 + 10001 + i) for d in range(pick(q(), [0.05, 0.12, 0.21, 0.62])): v = 0.0 if q() < 0.038 else math.floor( b[2] * math.exp((q() + q() + q() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 DATASET.append((sub_id, b[0], d + 1, household, satisfied, v, tier(v))) q() def split_parts(dataset, p): n = len(dataset) return [dataset[i * n // p:(i + 1) * n // p] for i in range(p)] def map_row(row, key_fn): return (key_fn(row), (row[5], 1)) # intermediate value: (total, count) def combine(a, b): return (a[0] + b[0], a[1] + b[1]) def run(parts, key_fn, use_combiner): """Map -> (local reduce, if any) -> shuffle -> reduce. The second number returned is the number of records moved in the shuffle.""" moved = [] for p in parts: pairs = [map_row(s, key_fn) for s in p] if use_combiner: local = {} for k, v in pairs: local[k] = combine(local[k], v) if k in local else v pairs = sorted(local.items(), key=lambda x: str(x[0])) moved += pairs bucket = {} for k, v in moved: bucket[k] = combine(bucket[k], v) if k in bucket else v return {k: (v[0] / v[1], v[1]) for k, v in bucket.items()}, len(moved) P = split_parts(DATASET, PARTS) result, moved = run(P, lambda s: s[1], False) print(f"seed {SEED}; {len(DATASET)} rows, {PARTS} source parts " f"{[len(x) for x in P]}") print(f"{'region':<10}{'rows':>7}{'average m3':>14}") for b in sorted(result): print(f"{b:<10}{result[b][1]:>7}{result[b][0]:>14.3f}") print(f"east - north = {result['east'][0] - result['north'][0]:.3f} m3; " f"records moved in the shuffle {moved}") ``` ``` seed 20260218; 3199 rows, 8 source parts [399, 400, 400, 400, 400, 400, 400, 400] region rows average m3 center 600 22.396 east 557 24.674 north 921 20.210 south 645 16.602 west 476 13.778 east - north = 4.463 m3; records moved in the shuffle 3199 ``` The dataset was spread across eight parts and was never brought together as a whole anywhere, yet the anchor of the course did not change: the east average is 24.674 m³, the north 20.210 m³, the difference 4.463 m³. When the model is built correctly, dividing does not move the answer; what it moves is the records carried to produce the answer. In this first run the records moved are 3,199, that is, the whole dataset. Map produced one pair for every row and all of the pairs went to the reducers. Reduce then ran and gave five numbers. For five numbers, 3,199 records were moved. What the intermediate value is turns out to be an invisible decision at this point. The map function does not send the reading alone; it puts a 1 next to it and sends a **(total, count)** pair. If a single number were sent, the reduce function would have to take the average of averages, and as soon as the part sizes were unequal the result would drift. Sending a pair does not double the records moved — the number of records is the same, the inside of the record widens — but it makes the reduce function associative. The third section of the lesson measures the case where this property is absent. #### Records Moved in the Shuffle ```python # transfer.py — the same aggregation is run in two orders: the order that shuffles first and # then reduces, and the order that reduces locally first and then shuffles. The measure is the # number of records moved in the shuffle. KEYS = (("region", lambda s: s[1]), ("region + period", lambda s: (s[1], s[2])), ("region + tier", lambda s: (s[1], s[6])), ("subscriber_id", lambda s: s[0])) print(f"{'grouping key':<18}{'keys':>9}{'shuffle first':>15}" f"{'reduce first':>14}{'gain':>9}{'same result':>13}") for label, f in KEYS: a, ta = run(P, f, False) b, tb = run(P, f, True) same = all(abs(a[k][0] - b[k][0]) < 1e-9 for k in a) print(f"{label:<18}{len(a):>9}{ta:>15}{tb:>14}{ta / tb:>8.2f}x" f"{('yes' if same else 'no'):>13}") print(f"records sent by the combiner are the distinct key count per part; " f"{PARTS} parts and 5 regions make {PARTS * 5} records") ``` ``` grouping key keys shuffle first reduce first gain same result region 5 3199 40 79.97x yes region + period 15 3199 120 26.66x yes region + tier 18 3199 137 23.35x yes subscriber_id 1260 3199 1265 2.53x yes records sent by the combiner are the distinct key count per part; 8 parts and 5 regions make 40 records ``` The order decision does not move the result, it moves the transfer. In all four rows the last column says "yes": shuffling first and reducing afterward and reducing locally first and shuffling afterward give the same five numbers. What changes is the records that pass through the shuffle, and on the region key they fall from 3,199 to 40 — **79.97 times**. Where the forty comes from is a one-line product: eight parts, five regions in each part. Local reduction brings all the rows in a part down to the number of keys in that part; the records moved now come not from the size of the dataset but from **the number of parts times the number of distinct keys within a part**. Even if the dataset grew a thousand times, the records moved on the region key would stay at forty. The last row draws the limit. The subscriber key has 1,260 distinct keys and the row count is 3,199; a subscriber has at most three rows. Local reduction brings two and a half rows per part down to one on average and the gain falls to 2.53 times. The combiner's gain comes from **the ratio of the number of keys to the number of rows**; as the number of keys approaches the number of rows, the combiner finds nothing to carry. The two middle rows show how this ratio works. When period is added to region, the number of keys rises from five to fifteen and the records moved come to exactly eight times fifteen, that is, 120: all fifteen keys are present in all eight parts. When tier is added, the number of keys is eighteen, but the records moved are 137, not 144. The shortfall of seven records is not a gain but a sparsity: two of the eighteen combinations of region and tier hold only a few rows in the whole dataset and do not appear at all in seven of the eight parts. The records moved are the sum of **the number of keys actually seen per part**, not of the number of keys, and this number is known only by running. #### Two Passes and the Computation That Does Not Enter the Model ```python # passes.py — some computations do not fit a single shuffle, and some do not fit at all. # First the two-pass computation, then whether the reduce function is associative is tested: # the same computation is run with seven different part counts. import statistics subs, t1 = run(P, lambda s: s[0], True) # pass 1: key is subscriber REGION_OF = {s[0]: s[1] for s in DATASET} second = [(REGION_OF[k], (v[0], 1)) for k, v in subs.items()] P2 = split_parts(second, PARTS) # pass 2: key is region t2 = sum(len({k for k, _ in p}) for p in P2) mean2 = {} for p in P2: for k, v in p: mean2[k] = combine(mean2[k], v) if k in mean2 else v single, t_single = run(P, lambda s: s[1], True) print(f"{'region':<10}{'single pass: row-weighted':>28}" f"{'two passes: subscriber average':>32}{'difference':>12}") for b in sorted(single): a2 = mean2[b][0] / mean2[b][1] print(f"{b:<10}{single[b][0]:>28.3f}{a2:>32.3f}{a2 - single[b][0]:>12.3f}") print(f"single pass moves {t_single} records; two passes move {t1} + {t2} = {t1 + t2} records") medians = [] print(f"{'parts':<8}{'total m3':>12}{'largest':>10}" f"{'median of part medians':>32}") for p in (2, 3, 4, 5, 8, 16, 32): splits = split_parts(DATASET, p) m = statistics.median([statistics.median([x[5] for x in y]) for y in splits]) medians.append(m) print(f"{p:<8}{sum(sum(x[5] for x in y) for y in splits):>12.2f}" f"{max(max(x[5] for x in y) for y in splits):>10}{m:>32.4f}") bucket_count = sum(len({x[5] for x in p}) for p in P) print(f"true median {statistics.median([s[5] for s in DATASET])} m3; seven part " f"counts give {len(set(medians))} distinct values, spread " f"{max(medians) - min(medians):.4f} m3") print(f"readings have two decimals: {len({s[5] for s in DATASET})} distinct values; count " f"combiner moves {bucket_count} records, gain {len(DATASET) / bucket_count:.2f}x") ``` ``` region single pass: row-weighted two passes: subscriber average difference center 22.396 22.483 0.087 east 24.674 24.814 0.140 north 20.210 20.351 0.141 south 16.602 16.728 0.127 west 13.778 13.935 0.158 single pass moves 40 records; two passes move 1265 + 40 = 1305 records parts total m3 largest median of part medians 2 63060.63 52.47 18.9900 3 63060.63 52.47 18.9700 4 63060.63 52.47 18.9250 5 63060.63 52.47 18.8400 8 63060.63 52.47 18.9725 16 63060.63 52.47 18.8900 32 63060.63 52.47 19.0575 true median 18.98 m3; seven part counts give 7 distinct values, spread 0.2175 m3 readings have two decimals: 1788 distinct values; count combiner moves 2839 records, gain 1.13x ``` The first table is the hidden decision of this lesson. "Regional average" looks like a single question, yet it is two separate computations. In the single pass the key is the region and every row carries equal weight: east 24.674 m³. In the two passes the reduction is first by the subscriber key, then the subscriber results are reduced by the region key, and every subscriber carries equal weight: east 24.814 m³. The difference is 0.140 m³ and in the same direction in all five regions. Both are defensible; when which one was chosen is not written down, the number in the report is not a finding but a choice. The cost is in the table too. The single pass moves 40 records, the two passes 1,265 plus 40, a total of 1,305 records — **thirty-three times**. As the number of passes grows, the records moved multiply, because the output of the intermediate pass becomes the input of the next pass and that output is as large as the number of keys. The second table draws the limit of entering the model. The sum and the maximum do not budge across seven different part counts: 63,060.63 m³ and 52.47 m³. The reduce functions are associative and commutative over partial results, so the splitting stays invisible. The median, on the other hand, gives **seven distinct values** across the seven part counts; the spread is 0.2175 m³ and none of them equals the true median, 18.98 m³. The median of part medians is not even an approximation; it is a number that depends on the part count. The last line gives the price of the way out. When the intermediate value of the median is a **count** instead of the median, it enters the model, but it does not save the day on this dataset: because the readings are written with two decimals there are 1,788 distinct values, and the count combiner moves 2,839 records, that is, a gain of 1.13 times. The number of decimals is a display decision and here it determines the length of the shuffle: the same readings written with one decimal would make the combiner work. #### Summary - The three functions ran and, although the dataset was spread across eight parts, the anchor did not change: east 24.674 m³, north 20.210 m³, difference 4.463 m³. - The order decision moves the transfer, not the result: on the region key shuffling first moves 3,199 records, reducing locally first moves 40, and the gain is 79.97 times. - The records a combiner moves are the product of the part count and the number of distinct keys in a part; on the subscriber key, 1,260 keys cut the gain to 2.53 times. - The same question gives two values with two keys: the east average is 24.674 m³ weighted by rows and 24.814 m³ as the average of subscriber averages, and the two passes move 1,305 records instead of 40. - A computation does not enter the model if the reduce function is not associative: the sum and the maximum come out the same across seven part counts, the median of part medians gives seven different values, and its spread is 0.2175 m³. - When the intermediate value of the median is turned into a count it enters the model, but with readings of two decimals there are 1,788 distinct values and the gain falls to 1.13 times. #### Next Step This lesson counted the records moved and showed that the transfer depends on the order, but it never asked who runs the three functions. In a real job the parts are defined somewhere, tasks are distributed, a failed task is retried, and the results are collected; none of these processes rows and all of them are added per job. The next lesson counts the **coordination overhead item** of data frame processing on a cluster: into how many stages a chain of operations is split, how many items are added per stage, how many a retry raises this number to, and at which job size the number of items exceeds the number of rows processed. ### Distributed Analysis Engines Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/distributed-analysis-engines Counting the coordination overhead items that data frame processing on a cluster adds per job: the three wide transformations of an eight-operation chain draw four stage boundaries and amount to 100 coordination overhead items over eight partitions; none of the items processes a row and their number is independent of job size. Retry is not the dominant item: at a 2% failure rate it raises the item count by 0-2%, at a 15% failure rate by 8-23%. The measure is the number of rows per coordination overhead item: 61.5 rows at 3,199 rows, 31,990 rows at 3,199,000 rows, and the ratio saturates at 32,878. A target of 10,000 rows per item is met at 520,000 rows in the four-stage chain and at 260,000 rows in the two-stage chain. The previous lesson wrote the three functions and counted the records moved in the shuffle, but it did not ask who runs the functions. When a data frame chain is written on a real cluster, the person who writes the chain orders the operations; the engine splits that ordering into **stages**, defines the partitions for each stage, distributes the tasks, retries a failed task, and collects the results. None of this work processes a row and none of it is written in the chain. The person who writes the chain thinks they have written a query; the engine derives a task schedule from that query, and the length of the schedule comes not from the length of the query but from the points at which the query has to redistribute the data. This is the measure of the lesson: **the coordination overhead item** — countable items that are added per job and do not grow with job size. The second number next to it is how many rows fall to one coordination overhead item. Distributed systems theory, partitioning, and coordination were established in the System Design and Distributed Systems and the DevOps and Platform Engineering curricula and are not repeated here; the only thing written is the item that is counted. - **AS13.** No distributed processing engine is installed and no product name is written. The chain, the stage, and the item are modeled with the standard library. **It is a model.** - **AS14.** A transformation is **narrow** if it ends within a partition (filter, select, derive), or **wide** if it needs a shuffle (group, sort, join). The stage boundary is drawn at a wide transformation. - **AS15.** Coordination overhead items are of four kinds: partition definition, task dispatch, task result, and stage collection. The first three are counted as many as the number of partitions per stage, the last one once per stage. - **AS16.** The probability that a task fails is an assumption and is tested with two values: 0.02 and 0.15. At most three tries are made; each retry adds two items. The generator is written in-house and the seed is 20260218. - **AS17.** The number of partitions is derived from the job size: part size 400,000 rows, at least four partitions. Both are decisions. - **AS18.** The acceptability criterion is a decision: at least 10,000 rows must fall to one coordination overhead item. #### How Many Stages the Chain Splits Into ```python # chain.py — a data frame chain is split into stages. MODEL: no distributed engine is # installed; the operations of the chain are marked narrow (ends within a partition) and # wide (needs a shuffle) and the stage boundary is drawn at a wide transformation. CHAIN = [("filter: period 1-3", "narrow"), ("derive: tariff tier", "narrow"), ("group: subscriber", "wide"), ("derive: subscriber average", "narrow"), ("filter: average > 10 m3", "narrow"), ("group: region", "wide"), ("sort: average", "wide"), ("first 3", "narrow")] PARTITIONS = 8 def to_stages(chain): stages, current = [], [] for label, kind in chain: current.append((label, kind)) if kind == "wide": stages.append(current) current = [] if current: stages.append(current) return stages A = to_stages(CHAIN) print(f"{len(CHAIN)} operations, {sum(1 for _, t in CHAIN if t == 'wide')} wide, " f"{len(A)} stages, {PARTITIONS} partitions") print(f"{'stage':<7}{'operations':>12}{'boundary':>10} {'content'}") for i, a in enumerate(A, 1): print(f"{i:<7}{len(a):>12}{a[-1][1]:>10} {' -> '.join(label for label, _ in a)}") # Coordination overhead item: in each stage, for each partition, a partition definition, # a task dispatch, and a task result; for each stage, one stage collection item. ITEMS = (("partition definition", "partition"), ("task dispatch", "partition"), ("task result", "partition"), ("stage collection", "stage")) print(f"{'item kind':<22}{'scale':>10}{'stages':>7}{'partitions':>11}{'items':>7}") total = 0 for label, scale in ITEMS: n = len(A) * PARTITIONS if scale == "partition" else len(A) total += n print(f"{label:<22}{scale:>10}{len(A):>7}{PARTITIONS if scale == 'partition' else '-':>11}{n:>7}") print(f"{'total':<22}{'':>10}{'':>7}{'':>11}{total:>7}") ``` ``` 8 operations, 3 wide, 4 stages, 8 partitions stage operations boundary content 1 3 wide filter: period 1-3 -> derive: tariff tier -> group: subscriber 2 3 wide derive: subscriber average -> filter: average > 10 m3 -> group: region 3 1 wide sort: average 4 1 narrow first 3 item kind scale stages partitions items partition definition partition 4 8 32 task dispatch partition 4 8 32 task result partition 4 8 32 stage collection stage 4 - 4 total 100 ``` The person who wrote the chain wrote eight operations; the engine built four stages. The number of stages is not written in the chain; it derives from the three wide transformations and is invisible when the chain is read. That the three operations of the second stage sit together is also an engine decision: the two narrow transformations work on the output of the shuffle before them, so no separate stage is needed. Whether a transformation counts as narrow or wide is not a given fact either; it is the result of the partitioning decision. The third operation of the chain is grouping by the subscriber key; if the dataset is partitioned by the subscriber key, all the rows of a subscriber are already in the same partition and the grouping needs no shuffle, that is, it is narrow. The same transformation is wide in a dataset partitioned by the period key. The four stages in the table rest on the assumption that the dataset is not partitioned by the subscriber key; when the partitioning key changes, the number of stages falls to three and the coordination overhead items fall by the same ratio. **The number of stages is the number that the chain and the partitioning give together, not the number of the chain alone.** The lower table gives the measure of the lesson. Four stages and eight partitions amount to **100 coordination overhead items** across four item kinds. Not one of these hundred items reads a row, adds a value, or builds a group. All of them are **around** the job, and their existence is written neither in the chain nor in the query. The item count is entirely independent of job size: the same chain asks for a hundred items at three thousand rows and at three billion rows. The four item kinds also need to be told apart, because three scale with the number of partitions and one with the number of stages. The **partition definition** is the determination of which rows a partition is responsible for in a stage. The **task dispatch** is the handing of that definition to an executor. The **task result** is the executor's reporting that it has finished and the recording of its output. These three are counted separately for each partition and multiplied by the number of partitions. The **stage collection** is seeing that all the partitions of the stage have finished and opening the next stage; there is one per stage and it is unaffected by the number of partitions. Ninety-six of the hundred items come from the first three kinds, four from the last. #### Retry Is Not the Dominant Item ```python # retry.py — task failure and the retry item. The generator is written in-house and the seed # is visible; the output is the same from run to run. MODEL: the probability that a task # fails is an assumption and is tested with two values. SEED, M32, MAX_TRIES = 20260218, 0xFFFFFFFF, 3 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def item_count(stage, partition): return stage * partition * 3 + stage def retries(stage, partition, q, seed): r, retried = rng(seed), 0 for _ in range(stage * partition): for _ in range(MAX_TRIES - 1): if r() < q: retried += 1 else: break return retried print(f"at most {MAX_TRIES} tries; each retry adds 2 items " f"(re-dispatch and re-result)") print(f"{'partitions':<12}{'tasks':>7}{'base items':>12}{'q=0.02 extra':>14}" f"{'q=0.02 rise':>13}{'q=0.15 extra':>14}{'q=0.15 rise':>13}") for b in (4, 8, 16, 64, 256): g, base = len(A) * b, item_count(len(A), b) e1 = 2 * retries(len(A), b, 0.02, SEED + 91000 + b) e2 = 2 * retries(len(A), b, 0.15, SEED + 91000 + b) print(f"{b:<12}{g:>7}{base:>12}{e1:>14}{(base + e1) / base:>12.2f}x" f"{e2:>14}{(base + e2) / base:>12.2f}x") ``` ``` at most 3 tries; each retry adds 2 items (re-dispatch and re-result) partitions tasks base items q=0.02 extra q=0.02 rise q=0.15 extra q=0.15 rise 4 16 52 0 1.00x 12 1.23x 8 32 100 0 1.00x 8 1.08x 16 64 196 4 1.02x 24 1.12x 64 256 772 6 1.01x 82 1.11x 256 1024 3076 40 1.01x 336 1.11x ``` Retry is the most visible item of distributed processing, and the table shows it as the smallest item. At a 2% failure rate the item count rises by at most 2%; even at a 15% failure rate the rise stays between 8 and 23%. That the 1.23 times in the first row is higher than the others is not a trend but the fluctuation of a small count of sixteen tasks — the same ratio settles at 1.11 times at 1,024 tasks. The dominant item is in the third column of the table. When the number of partitions rises from four to 256, the base item count grows from 52 to 3,076, nearly sixty times, and this growth comes not from failure but from **the partitioning decision**. Raising the number of partitions buys parallelism and, in the same motion, grows the coordination overhead item linearly. Retry is the noise of this table; the number that decides is the number of partitions. It also needs to be written what the item is not. A coordination overhead item is not a unit of work; a code change that speeds up the computation does not lower the item count, because the items are around the computation, not inside it. Even if the work of a partition becomes ten times faster, that partition still counts one partition definition, one task dispatch, and one task result. There are three decisions that lower the item count and all three are taken in the chain itself: reducing the number of stages, reducing the number of partitions, and aligning the grouping key with the partitioning key. #### Rows per Item ```python # scale.py — the item count is independent of job size, the row count is not. # The measure: how many rows fall to one coordination overhead item. TARGET is a decision. import math TARGET, PART_SIZE, MIN_PARTITIONS = 10000, 400000, 4 CHAIN_B = [("filter: period 1-3", "narrow"), ("derive: tariff tier", "narrow"), ("group: region", "wide"), ("first 3", "narrow")] B = to_stages(CHAIN_B) def partition_count(n): return max(MIN_PARTITIONS, math.ceil(n / PART_SIZE)) def items(n, stage): b = partition_count(n) return item_count(stage, b) + 2 * retries(stage, b, 0.02, SEED + 91000 + b) print(f"chain A {len(A)} stages, chain B {len(B)} stages; part size " f"{PART_SIZE} rows, at least {MIN_PARTITIONS} partitions, target {TARGET} rows/item") print(f"{'job size':<14}{'partitions':>12}{'A items':>9}{'A rows/item':>15}" f"{'B items':>9}{'B rows/item':>15}") for n in (3199, 31990, 319900, 3199000, 31990000, 319900000): ka, kb = items(n, len(A)), items(n, len(B)) print(f"{n:<14}{partition_count(n):>12}{ka:>9}{n / ka:>15.1f}{kb:>9}" f"{n / kb:>15.1f}") threshold = {} for label, stage in (("A", len(A)), ("B", len(B))): n = 1000 while n / items(n, stage) < TARGET and n < 10 ** 9: n += 1000 threshold[label] = n print(f"smallest job size that reaches the target: chain A {threshold['A']} rows, " f"chain B {threshold['B']} rows; halving the stages shrank the threshold " f"{threshold['A'] / threshold['B']:.1f} times") ``` ``` chain A 4 stages, chain B 2 stages; part size 400000 rows, at least 4 partitions, target 10000 rows/item job size partitions A items A rows/item B items B rows/item 3199 4 52 61.5 26 123.0 31990 4 52 615.2 26 1230.4 319900 4 52 6151.9 26 12303.8 3199000 8 100 31990.0 50 63980.0 31990000 80 976 32776.6 490 65285.7 319900000 800 9730 32877.7 4866 65741.9 smallest job size that reaches the target: chain A 520000 rows, chain B 260000 rows; halving the stages shrank the threshold 2.0 times ``` The table shows when coordination becomes invisible. In the 3,199 rows of the fictional dataset, 61.5 rows fall to one coordination overhead item: the job is much smaller than the work that sets it up. In a dataset a thousand times larger 31,990 rows fall and the ratio saturates — the last two rows give 32,777 and 32,878, because the number of partitions also grows with the job size and the ceiling comes from the share of the part size that falls to each stage. The saturation number is not made up; it derives from two decisions. The number of partitions is the job size divided by the part size, and the item count is stages times partitions times three; when the two cancel each other, the rows falling to one item settle at the part size divided by three times the number of stages. At four stages this is 400,000 over 12, that is, 33,333, and it appears in the table as 32,878 — the difference comes from the stage collection items and the retries. Without enlarging the part size or reducing the number of stages, this ceiling cannot be exceeded; beyond a certain point, enlarging the job size does not move the ratio at all. When the acceptability criterion is written as 10,000 rows per item, the four-stage chain meets it at 520,000 rows and the two-stage chain at 260,000 rows. **Halving the number of stages halves the threshold exactly.** The number of stages looks like an engine detail, yet it is a decision given in the way the chain is written: every wide transformation draws a stage boundary and that boundary produces items for all the partitions. It also needs to be written that the two chains do not ask the same question. Chain A first groups by the subscriber key, chain B directly by the region key; the difference measured in the previous lesson comes from here — the east average moves 0.140 m³ between 24.814 m³ and 24.674 m³. Reducing stages is not free; what it reduces is sometimes the question itself. This lesson must not be confused with the threshold in the Scale and Performance topic of the Data Analysis with Python course. There the same question was asked in terms of the **unit of work**, and the size at which distribution turned into a gain was measured as 10,833 rows on a balanced key; what was measured was performance on the analyst's own machine. Here what is measured is not the unit but the **item**, and the question asked is not when distribution becomes cheap but when coordination stays invisible. The two thresholds belong to different questions and cannot be written in place of each other. Distributing a job has to clear both thresholds: a job that does not clear the gain threshold slows down when distributed, and a job that does not clear the item threshold runs but is spent more on its surroundings than on the job itself. #### Summary - The three wide transformations of an eight-operation chain draw four stage boundaries; the number of stages is not written in the chain and is an engine decision. - Four stages and eight partitions amount to 100 coordination overhead items across four item kinds; none of the items processes a row and their number is independent of job size. - Retry is not the dominant item: at a 2% failure rate the rise is at most 2%, at a 15% failure rate it is between 8 and 23%. - The dominant item is partitioning: when the partitions rise from four to 256, the base item count rises from 52 to 3,076. - The rows that fall to one coordination overhead item are 61.5 in a 3,199-row job and 31,990 in a 3,199,000-row job, and the ratio saturates at 32,878. - A target of 10,000 rows per item is met at 520,000 rows in the four-stage chain and at 260,000 rows in the two-stage chain; halving the stages halves the threshold but also changes the question asked. #### Next Step This lesson counted the item around the computation, but it never opened the data the computation reads. When a stage starts, the partitions read their shares from somewhere, and the **layout** of that place determines how many values will be read. The next lesson takes up the columnar file layout and ties the measure to the values read: how many values the row group headers and column boundaries let you skip, how many values the header itself makes you read, how many values the same query reads in three layouts, and in which direction the row group size moves this number. ### Columnar Storage Formats Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/columnar-storage-formats Measuring the number of values read by a file layout suited to analytical queries: the 3,199 rows and 7 columns of the fictional dataset amount to 22,393 values; the same query reads 22,393 values in the row-ordered flat layout, 6,398 in the column-bounded layout, and 2,069 in the row-grouped layout with headers, and five of the seven row groups are never opened. The row group size is a write decision and costs in both directions: in a 32-row group the headers make you read 300 values, in a single 3,199-row group nothing is skipped; the size that reads the fewest values is 64 rows and gives 19.07 times with 1,174 values. Which headers are read is the second decision and moves the best size from 64 to 128. The gain of the column boundary melts with the number of columns the query asks for: it falls to 1.00 times for a query asking for seven columns, and row group skipping gives 4.92 times. The previous lesson counted the coordination overhead item around the computation and wrote how many items a stage adds, but it never opened what the partitions read when a stage starts. A partition reads its share from a file, and the **layout** of that file determines how many values must be touched to produce the same answer. The measure of this lesson is **the number of values read**, and what is measured is not an engine but a file. The distinction matters: the Scale and Performance topic of the Data Analysis with Python course used the same unit of measure to measure the **read path** — when an engine applies the filter, when it materializes, which column the dataset is sorted by. Here the read path is fixed and the write order is fixed too; the only thing that moves is **the layout of the file itself**: whether the values are separated by column boundaries, whether they are divided into row groups, whether a row group header exists and how much space that header takes. - **AS19.** The dataset is inherited from the Data Collection and Preparation, Data Analysis with Python, and Statistical Analysis and Exploratory Data Analysis courses and **is fictional**: 3,199 rows, 1,329 subscribers, three periods, five regions, tariff tiers 10/25/40 m³. It is written with seven columns and amounts to 22,393 values in total. **It is a model.** - **AS20.** No file format product is used; the three layouts are modeled in memory and every value read is counted. **It is a model.** - **AS21.** The write order is fixed: the file is written in the subscriber order the source gave. The effect of the sorting decision on skipping was measured in the Scale and Performance topic of the Data Analysis with Python course and is **not repeated** here. - **AS22.** A row group header carries three values per column: the minimum, the maximum, and the row count. The header is written to the file and reading it is counted too. - **AS23.** A row group is **not opened** if the filter range falls entirely outside the minimum and maximum values of that group. - **AS24.** The query is fixed: the total consumption in a given subscriber range. It asks for two columns and its answer is 8,869.88 m³. #### Three Layouts, One Query ```python # layout.py — the same FICTIONAL dataset is written in three file layouts and a single query is # run in all three. MODEL: no file format product is used; the layouts are modeled in memory # and the measure is the number of values read to produce the answer. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("center", 0.18, 23)] COLUMN = ("subscriber_id", "region", "period", "household", "satisfaction", "consumption_m3", "tier") def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tier(v): return "0-10" if v <= 10 else "10-25" if v <= 25 else "25-40" if v <= 40 else "over 40" DATASET = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfied = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id, q = 10001 + i, rng(SEED + 7000 + 10001 + i) for d in range(pick(q(), [0.05, 0.12, 0.21, 0.62])): v = 0.0 if q() < 0.038 else math.floor( b[2] * math.exp((q() + q() + q() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 DATASET.append((subscriber_id, b[0], d + 1, household, satisfied, v, tier(v))) q() N, S = len(DATASET), len(COLUMN) LOW, HIGH = 10601, 10800 # query: total consumption in this subscriber range selected = [s for s in DATASET if LOW <= s[0] <= HIGH] print(f"seed {SEED}; {N} rows, {S} columns, {N * S} values") print(f"file was written in the order the source gave: subscriber {min(s[0] for s in DATASET)} - " f"{max(s[0] for s in DATASET)}") print(f"query range {LOW}-{HIGH}: {len(selected)} rows match, correct answer " f"{sum(s[5] for s in selected):.2f} m3") ``` ``` seed 20260218; 3199 rows, 7 columns, 22393 values file was written in the order the source gave: subscriber 10001 - 11399 query range 10601-10800: 453 rows match, correct answer 8869.88 m3 ``` The boundary of the dataset is 22,393 values. The query concerns 453 rows of these values and asks for two columns; that is, the values the answer genuinely needs number 906. The ratio between the two is the frame of all the tables of this lesson: the better a layout is, the closer the number of values read gets to that 906, and no layout can go below it. ```python # reading.py — the same query is run in three layouts. Measure: number of values read. # D1 row-ordered flat layout, D2 column-bounded single row group, # D3 row-grouped and headed columnar layout. REQUESTED = ("subscriber_id", "consumption_m3") FILTER, HEADER_VALUE = "subscriber_id", 3 # minimum, maximum, row count def row_groups(r): return [DATASET[i:i + r] for i in range(0, N, r)] def d1(): return N * S, N, "all fields of a row contiguous" def d2(): return N * len(REQUESTED), N, "only the requested columns" def d3(r, all_headers=False): g = row_groups(r) header = HEADER_VALUE * len(g) * (S if all_headers else 1) opened = [k for k in g if not (max(x[0] for x in k) < LOW or min(x[0] for x in k) > HIGH)] data = sum(len(k) for k in opened) * len(REQUESTED) return header + data, sum(len(k) for k in opened), len(g) - len(opened) print(f"query asks for {len(REQUESTED)} columns: {', '.join(REQUESTED)}; " f"header is {HEADER_VALUE} values per row group") print(f"{'layout':<40}{'values read':>14}{'rows read':>14}{'ratio':>8}") t1, s1, _ = d1() t2, s2, _ = d2() t3, s3, skipped = d3(512) print(f"{'D1 row-ordered flat layout':<40}{t1:>14}{s1:>14}{t1 / t1:>7.2f}x") print(f"{'D2 column-bounded, single row group':<40}{t2:>14}{s2:>14}{t1 / t2:>7.2f}x") print(f"{'D3 row groups (512) with headers':<40}{t3:>14}{s3:>14}{t1 / t3:>7.2f}x") print(f"in D3, {skipped} of the {len(row_groups(512))} row groups were never opened") ``` ``` query asks for 2 columns: subscriber_id, consumption_m3; header is 3 values per row group layout values read rows read ratio D1 row-ordered flat layout 22393 3199 1.00x D2 column-bounded, single row group 6398 3199 3.50x D3 row groups (512) with headers 2069 1024 10.82x in D3, 5 of the 7 row groups were never opened ``` The three layouts give the same answer and the values read fall from 22,393 to 2,069 — **10.82 times**. The gain comes from two separate layout properties and each row of the table adds one of them. The first property is the **column boundary**. In the row-ordered flat layout the seven fields of a row are written side by side; to read two fields you have to pass over all seven. In a file drawn with column boundaries each column sits in its own contiguous area and only two of them are read: 22,393 values fall to 6,398. This gain comes not from a query or an engine but from how the file was written. The second property is **the row group and the header**. The file is divided into seven groups of 512 rows, and for each group the minimum, maximum, and row count of the subscriber column are written. If the query range does not touch the boundaries of a group, that group is not opened; five of the seven groups were not opened and the rows read fell from 3,199 to 1,024. The cost of reading the headers is 21 values, and what it saved is 4,350 values. The third column shows the remaining share of skipping. The query wants 453 rows and the rows opened are 1,024; the 571 rows in between are read, and the filter eliminates them one by one. The reason is that skipping works **at the grain of the row group**: if a group touches the range with even a single row, it is opened in its entirety. The fineness of skipping is determined by the row group size, and that is the subject of the next section. #### Row Group Size Is a Write Decision ```python # row_group.py — the row group size is a WRITE decision. A small row group gives fine skipping # and many headers, a large row group gives coarse skipping and few headers. # The write order is fixed: the subscriber order the source gave. print(f"{'row group':<14}{'groups':>6}{'header':>8}{'skipped':>9}" f"{'opened rows':>14}{'data':>7}{'total':>8}{'ratio':>8}{'all headers':>12}") best = None for r in (32, 64, 128, 256, 512, 1024, 3199): g = row_groups(r) total, opened, skipped = d3(r) all_hdr = d3(r, True)[0] header = HEADER_VALUE * len(g) print(f"{r:<14}{len(g):>6}{header:>8}{skipped:>9}{opened:>14}" f"{opened * len(REQUESTED):>7}{total:>8}{t1 / total:>7.2f}x{all_hdr:>12}") if best is None or total < best[1]: best = (r, total) all_best = min(((r, d3(r, True)[0]) for r in (32, 64, 128, 256, 512, 1024, 3199)), key=lambda x: x[1]) print(f"if only the filter column header is read, the best row group size is " f"{best[0]}: {best[1]} values, {t1 / best[1]:.2f} times") print(f"if the headers of all columns are read, the best size is {all_best[0]}: {all_best[1]} values, " f"{t1 / all_best[1]:.2f} times; same file, two read decisions, two different bests") # The gain depends on how many columns the query asks for. print(f"{'requested cols':<15}{'D1':>8}{'D2':>8}{'D2 ratio':>10}" f"{'D3 (128)':>10}{'D3 ratio':>10}") for k in (1, 2, 3, 5, 7): REQUESTED = COLUMN[:k] a, _, _ = d2() b = d3(128)[0] print(f"{k:<15}{t1:>8}{a:>8}{t1 / a:>9.2f}x{b:>10}{t1 / b:>9.2f}x") REQUESTED = ("subscriber_id", "consumption_m3") ``` ``` row group groups header skipped opened rows data total ratio all headers 32 100 300 85 480 960 1260 17.77x 3060 64 50 150 42 512 1024 1174 19.07x 2074 128 25 75 20 640 1280 1355 16.53x 1805 256 13 39 10 768 1536 1575 14.22x 1809 512 7 21 5 1024 2048 2069 10.82x 2195 1024 4 12 3 1024 2048 2060 10.87x 2132 3199 1 3 0 3199 6398 6401 3.50x 6419 if only the filter column header is read, the best row group size is 64: 1174 values, 19.07 times if the headers of all columns are read, the best size is 128: 1805 values, 12.41 times; same file, two read decisions, two different bests requested cols D1 D2 D2 ratio D3 (128) D3 ratio 1 22393 3199 7.00x 715 31.32x 2 22393 6398 3.50x 1355 16.53x 3 22393 9597 2.33x 1995 11.22x 5 22393 15995 1.40x 3275 6.84x 7 22393 22393 1.00x 4555 4.92x ``` The first table is the hidden decision of the lesson. The row group size is chosen when a file is written, not when a query is written, and it costs in both directions. In a small group skipping gets finer — in a 32-row group 85 of a hundred groups are skipped and only 480 rows are opened — but the headers make you read 300 values, nearly a third of the data share. In a large group the header gets cheap, only three values in a single 3,199-row group, but nothing is left to skip and the read rises to 6,401 values. The size that reads the fewest values is 64 rows: 1,174 values and **19.07 times** against 22,393. The top of the table is not sharp — all the sizes between 32 and 128 stay above 16 times — but the ends are expensive, and the two ends are expensive for two separate reasons. It needs to be written again that what is measured in this table is not an engine behavior: the write order is the same in all seven rows, the query is the same, the read path is the same. The only thing that changes is how many rows apart a header is written to the file. It also needs to be written where the ceiling is. The values the answer genuinely needed were 906; the best layout reads 1,174 values, that is, it is 1.30 times away from the best attainable. The remaining difference is made of two items: the header of 150 values and the 118 values of the 59 rows that fail to pass the filter. These two items cannot be zeroed together at any row group size, because one grows with the size and the other shrinks with it. The job of a file layout is not to meet the need of the query at zero cost but to find the size that keeps the sum of the two items the smallest. The last column adds the second decision. In some layouts the reader pulls only the header of the filter column; in others all the column headers of a row group sit as a single block and are read together. In the second case the header is seven times more expensive and the best size moves from 64 to 128: 1,805 values, 12.41 times. **The same file, two defensible read decisions, two different best sizes.** A layout has no single number called "the best row group size"; that number cannot be produced without writing which header is read. The second table shows that the gain depends on the query. The gain of the column boundary melts with the number of columns asked for and falls to 1.00 times when all seven columns are asked for: for a query that reads all the columns, the columnar layout is no different from the flat layout. Row group skipping does not melt, because skipping rows is independent of choosing columns; in the same row the ratio stays at 4.92 times. **The column boundary lets you skip the column that will not be read, the row group header the row that will not be read; the gains of the two are multiplied, and when one is zeroed the other continues.** This distinction determines which question will be asked when a file layout is chosen. Queries that read few columns and scan many rows gain from the column boundary; queries that read a given range gain from the row group header. A query that does neither — an export asking for all the columns and all the rows — gains from no layout: the column boundary gives 1.00 times, there is no row group to skip, and the read rises above the flat layout by the value of the headers. The layout decision is made before the query, and which query will be answered cheaply is determined by that decision. #### Summary - The 3,199 rows and 7 columns of the fictional dataset amount to 22,393 values; the values the query genuinely needs are two columns of 453 rows, that is, 906. - The same query reads 22,393 values in the row-ordered flat layout, 6,398 in the column-bounded layout, and 2,069 in the row-grouped layout with headers; five of the seven row groups are never opened and the gain is 10.82 times. - The row group size costs in both directions: in a 32-row group the headers make you read 300 values, in a single 3,199-row group nothing is skipped; the size that reads the fewest values is 64 and gives 19.07 times with 1,174 values. - Which headers are read is the second decision: when all the column headers are read, the best size moves to 128 and the gain falls to 12.41 times. - The gain of the column boundary melts with the number of columns asked for and falls to 1.00 times in a seven-column query; row group skipping keeps giving 4.92 times in the same query. #### Next Step Four lessons counted the items of distribution one by one: records moved, coordination overhead items, values read. All three were items that could be reduced and a decision was found in each. What remains is a serial fraction that cannot be reduced. Whatever percentage of a job is indivisible, however many units are added, the speedup cannot rise above a ceiling. The next lesson presses on this ceiling by increasing the number of units and writes with numbers when distribution is unnecessary; the same lesson closes the course and the Data Analytics curriculum. ### Limits of Parallel Processing Source: https://academia.sh/en/courses/visualization-reporting-and-analysis-at-scale/limits-of-parallel-processing The cost of distribution and when it is unnecessary are counted through the serial fraction and the speedup ceiling: of the 6,753 touches of the fictional report chain 310 are indivisible, the serial fraction is 4.59% and the ceiling is 21.78 times. At a serial fraction of 2% the ceiling is 50, at 30% it is 3.33 times, and even 4,096 units cannot pass the ceiling. The gain per added unit falls from 0.91 to 0.00011; after 256 units each new unit gains less than two thousandths. How the ceiling behaves with job size is an assumption decision: if the indivisible items are held fixed the ceiling rises to 206,388 times, if the coordination overhead item and the row group header grow with job size it saturates at 43.64 times. The course wrap-up gathers the source and read-back numbers of the eighteen lessons, and the curriculum wrap-up the six courses and 90 lessons of the Data Analytics curriculum in a single table. Four lessons counted the items of distribution one by one: the records moved in the shuffle, the coordination overhead item added per job, the values a file layout makes you read. All three could be reduced and each held two defensible decisions. This lesson measures the serial fraction, which cannot be reduced. Not every step of a job is divisible. The schema of the source is read once, the job plan is derived once, the result table is built and written once; these steps run once however many units are added, and do not shorten. The measure of the lesson is **the serial fraction** — the ratio of the indivisible part of the job to the whole job — and the second number derived from it is **the speedup ceiling**: the highest speedup, in times, reachable even as the number of units goes to infinity. - **AS25.** The report chain is built from the items the previous four lessons counted and **is fiction**: the row count is 3,199, the header value is 150 (from the best row group size of the fourth lesson), the coordination overhead item is 100 (from the four-stage, eight-partition chain of the third lesson), and the records moved in the shuffle are 40 (from the second lesson). **It is a model.** - **AS26.** The unit of work is a touch; raw milliseconds are not written and the number measured is **the speedup in times**. - **AS27.** Speedup is computed from the serial fraction $s$ and the number of units $p$ as $1 / (s + (1 - s) / p)$; as $p$ goes to infinity the ceiling becomes $1 / s$. - **AS28.** The divisible steps are assumed spread equally among the units. Partition imbalance was measured in the Scale and Performance topic of the Data Analysis with Python course and is ignored here; this assumption gives the ceiling **optimistically**. - **AS29.** Whether the indivisible items grow with job size is a decision, tested with two assumptions: a fixed series, and a series that grows by the formulas of the third and fourth lessons. - **AS30.** The number of units needed to reach 90% of the ceiling is found by scanning multiples of two. #### The Serial Fraction of the Job ```python # ceiling.py — the serial fraction of the job and the speedup ceiling. MODEL: the report # chain is split into steps, each step is given a number of touches and whether it is divisible; # the measure is the speedup in times, not raw time. N = 3199 STEPS = (("reading the source schema", 150, "no"), ("deriving the job plan and distributing tasks", 100, "no"), ("reading and filtering the rows", N, "yes"), ("local reduction", N, "yes"), ("records moved in the shuffle", 40, "yes"), ("final reduction (5 keys)", 5, "yes"), ("building and writing the single report table", 60, "no")) print(f"{'step':<46}{'touches':>9}{'divisible':>10}") for name, d, b in STEPS: print(f"{name:<46}{d:>9}{b:>10}") TOTAL = sum(d for _, d, _ in STEPS) SERIAL = sum(d for _, d, b in STEPS if b == "no") S = SERIAL / TOTAL print(f"total touches {TOTAL}, indivisible {SERIAL}, serial fraction " f"{100 * S:.2f} percent, speedup ceiling {1 / S:.2f} times") ``` ``` step touches divisible reading the source schema 150 no deriving the job plan and distributing tasks 100 no reading and filtering the rows 3199 yes local reduction 3199 yes records moved in the shuffle 40 yes final reduction (5 keys) 5 yes building and writing the single report table 60 no total touches 6753, indivisible 310, serial fraction 4.59 percent, speedup ceiling 21.78 times ``` The serial fraction is not a property of code but **a list of steps**. The three steps total 310 touches, 4.59% of the job of 6,753 touches. Each has its own reason for not being divided: the schema sits in a single place of the source, the job plan has to see all the partitions, the result table has to be a single output. 4.59% looks like a small number. Its counterpart is 21.78 times: however many units you divide this job into, the speedup cannot pass it. A serial fraction of 4.59% cuts off one twenty-second of the speedup from the start. #### Speedup Ceiling ```python # speedup.py — the same job is run with different serial fractions and an increasing number of # units. speedup(p) = 1 / (s + (1 - s) / p); the ceiling is 1 / s as p goes to infinity. UNITS = (1, 2, 4, 8, 16, 64, 256, 1024, 4096) def speedup(s, p): return 1.0 / (s + (1.0 - s) / p) def units_for_90(s): p = 1 while speedup(s, p) < 0.9 / s and p < 10 ** 7: p *= 2 return p SHARES = (0.02, S, 0.10, 0.30) print("units".ljust(8) + "".join(f"s={s:.4f}".rjust(10) for s in SHARES)) for p in UNITS: print(f"{p:<8}" + "".join(f"{speedup(s, p):>10.2f}" for s in SHARES)) print(f"{'ceiling':<8}" + "".join(f"{1 / s:>10.2f}" for s in SHARES)) print(f"{'p90':<8}" + "".join(f"{units_for_90(s):>10}" for s in SHARES)) ``` ``` units s=0.0200 s=0.0459 s=0.1000 s=0.3000 1 1.00 1.00 1.00 1.00 2 1.96 1.91 1.82 1.54 4 3.77 3.52 3.08 2.11 8 7.02 6.05 4.71 2.58 16 12.31 9.48 6.40 2.91 64 28.32 16.44 8.77 3.22 256 41.97 20.15 9.66 3.30 1024 47.72 21.35 9.91 3.33 4096 49.41 21.67 9.98 3.33 ceiling 50.00 21.78 10.00 3.33 p90 512 256 128 32 ``` The right column is the most instructive. When the serial fraction is 30%, 4,096 units give **3.33 times**: four thousand ninety-six units, three and a half times. With two units the same job speeds up 1.54 times; the last 4,094 units add only 1.79 times. A serial fraction of 30% means 70% of the job is divisible, which sounds like a job that can be distributed. The left column shows the same thing from the other side. When the serial fraction is 2% the ceiling is 50 times and 4,096 units give 49.41 times. The difference between 2% and 30% is not fifteen times, it is a **fifteen-times** difference in the ceiling: 50 against 3.33. The serial fraction rises linearly; the ceiling falls in inverse proportion. The last row gives the practical side. The number of units needed to reach 90% of the ceiling grows as the serial fraction gets smaller: at 30% 32 units are enough, at 2% 512 are needed. A small serial fraction gives a high ceiling, but it asks for more units to reach it. #### The Added Unit and the Ceiling Itself ```python # limit.py — as the ceiling is approached the times gained by an added unit melt away; and the # serial fraction changes with job size because the divisible items grow and the # indivisible items do not. print(f"serial fraction {100 * S:.2f} percent; ceiling {1 / S:.2f} times") print(f"{'units':<8}{'speedup':>10}{'rise over previous':>23}" f"{'per added unit':>22}{'distance to ceiling':>21}") prev_p, prev_h = 1, 1.0 for p in UNITS[1:]: h = speedup(S, p) print(f"{p:<8}{h:>10.2f}{h - prev_h:>23.3f}" f"{(h - prev_h) / (p - prev_p):>22.5f}{1 / S - h:>21.2f}") prev_p, prev_h = p, h import math FIXED = sum(d for _, d, b in STEPS if b == "no") def growing(n): """The coordination overhead item of lesson 03 and the header value of lesson 04 grow with job size.""" partition = max(8, math.ceil(n / 400000)) return 4 * partition * 3 + 4 + 3 * math.ceil(n / 64) + 60 print(f"{'job size':<12}{'divisible':>13}{'fixed serial':>13}{'fixed ceiling':>15}" f"{'growing serial':>16}{'growing ceiling':>17}{'p90':>9}") for n in (3199, 31990, 319900, 3199000, 31990000): div = 2 * n + 45 s1, s2 = FIXED / (div + FIXED), growing(n) / (div + growing(n)) print(f"{n:<12}{div:>13}{FIXED:>13}{1 / s1:>15.1f}{growing(n):>16}" f"{1 / s2:>17.2f}{units_for_90(s2):>9}") LARGEST = 31990000 saturation = (2 * LARGEST + 45 + growing(LARGEST)) / growing(LARGEST) print(f"the fixed-serial assumption makes the ceiling grow in direct proportion to job size; under the " f"growing-serial assumption the ceiling saturates at {saturation:.2f} times and 90 percent of it " f"is reached at 512 units") ``` ``` serial fraction 4.59 percent; ceiling 21.78 times units speedup rise over previous per added unit distance to ceiling 2 1.91 0.912 0.91222 19.87 4 3.52 1.604 0.80180 18.27 8 6.05 2.539 0.63466 15.73 16 9.48 3.421 0.42762 12.31 64 16.44 6.968 0.14517 5.34 256 20.15 3.704 0.01929 1.64 1024 21.35 1.202 0.00157 0.43 4096 21.67 0.323 0.00011 0.11 job size divisible fixed serial fixed ceiling growing serial growing ceiling p90 3199 6443 310 21.8 310 21.78 256 31990 64025 310 207.5 1660 39.57 512 319900 639845 310 2065.0 15157 43.21 512 3199000 6398045 310 20639.9 150115 43.62 512 31990000 63980045 310 206388.2 1500556 43.64 512 the fixed-serial assumption makes the ceiling grow in direct proportion to job size; under the growing-serial assumption the ceiling saturates at 43.64 times and 90 percent of it is reached at 512 units ``` The first table gives what the added unit gains. The second unit adds 0.91 times; each unit in the 4,096th unit group adds 0.00011 times. **After 256 units each added unit gains less than two thousandths** and the distance to the ceiling has fallen to 1.64 times. What follows is not a measurable gain, only an asymptotic approach to the ceiling. The second table is the hidden decision of the lesson. "The bigger the dataset, the more distribution gains" is true, but how much it gains depends on an assumption. If the indivisible items are held fixed, the ceiling grows in direct proportion to job size and rises to 206,388 times at 31,990,000 rows. Yet the third lesson measured that the coordination overhead item grows with the number of partitions and the fourth that the row group header grows with the number of row groups. When these two formulas are taken into account the ceiling **saturates at 43.64 times** and enlarging the job size no longer moves it. The ratio between the two numbers is 4,729 times, and both come from the same model, dataset and chain. The only difference is whether the indivisible items are assumed to grow with job size, an assumption no speedup report writes spontaneously. When distribution is unnecessary can be read from here too. In the 3,199 rows of the fictional dataset the ceiling appears as 21.78 times, but the third lesson counted that at this job size only 61.5 rows fall to one coordination overhead item and that the target of 10,000 rows per item is met at 520,000 rows; the first lesson wrote that with the chosen decision the dataset would stop fitting on a single machine at 22,525,682 rows. The three numbers point in the same direction: this dataset is not distributed. A high ceiling does not call for distribution; what calls for it is the job itself exceeding a single machine. #### Summary - The serial fraction is not a property of code but a list of steps: reading the schema, the job plan, and writing the single report table total 310 touches, 4.59% of the job of 6,753 touches. - A serial fraction of 4.59% pins the speedup ceiling to 21.78 times; at 2% the ceiling is 50, at 30% it is 3.33 times, and even 4,096 units cannot pass the ceiling. - Reaching the ceiling is not free either: 90% of it takes 32 units at a 30% serial fraction and 512 units at a 2% serial fraction. - The gain of the added unit melts away: the second unit adds 0.91 times, after 256 units each unit adds less than two thousandths, and the distance to the ceiling falls to 1.64 times. - The behavior of the ceiling with job size is an assumption decision: if the indivisible items are held fixed the ceiling rises to 206,388 times, if they grow with the formulas of the third and fourth lessons it saturates at 43.64 times; there are 4,729 times between the two numbers. - A high ceiling does not call for distribution: in the 3,199-row dataset, although the ceiling is 21.78 times, the item threshold is 520,000 and the volume threshold is 22,525,682 rows. #### Course Wrap-Up In each of the eighteen lessons two numbers stood side by side, what was in the data and what reached the reader, each read from its own lesson. | Lesson | Number in the source | Number read back | Difference | |---|---|---|---| | Chart 01 Chart Type Selection | east - north 4.463 m³ | 2.500 m³ from the area channel | -1.963 m³ | | Chart 02 Distribution Visualization | 2 peaks, outlier 45.47 m³ | 0 peaks in the box summary | 2 peaks lost | | Chart 03 Categorical Visualization | 39 pairs that must be told apart | 35 pairs in the stacked chart | 4 pairs, error 47.2% | | Chart 04 Relationship Visualization | 1,101 points | 309 filled cells | 792 points, 71.9% | | Chart 05 Time Series Visualization | trend -0.140 m³/period | -0.1347 m³/period | 0.0053, cost 12 periods | | Chart 06 Geographic Visualization | average 19.713 m³ | 18.484 m³ weighted by area | -1.228 m³ | | Chart 07 Misleading Visuals | north 20.210 m³ | 8.019 m³ on the truncated axis | -12.191 m³ | | Chart 08 Accessible Charts | 8 series | 4 series when tone is not read | 4 series, 0 with redundancy | | Reporting 01 Dashboard Design | 12 criteria | 9 criteria in the layout by decision | 3 criteria left out | | Reporting 02 Key Performance Indicators | coverage 60.948 | up to 93.766 by six routes | 32.818 points | | Reporting 03 Business Intelligence Tools | final value 63.055 | 36.042 in the pre-aggregated one | 27.013 points | | Reporting 04 Building Narrative with Data | 16 premises | 2-4 inferences when one step is missing | 11 wrongly assumed, 9 in favor | | Reporting 05 Report Reproducibility | 59 items to pin | 10 of 11 items move it | 36.042 ↔ 100.000 | | Scale 01 Big Data Concept | 286.0 bytes per row | threshold 22,525,682 rows | 7,508,560 ↔ 240,273,951 | | Scale 02 Map–Reduce Model | 3,199 records | 40 records in the shuffle | 79.97 times | | Scale 03 Distributed Analysis Engines | 8 operations | 4 stages, 100 items | 61.5 rows/item | | Scale 04 Columnar Storage Formats | 22,393 values | 1,174 values in the best layout | 19.07 times | | Scale 05 Limits of Parallel Processing | 6,753 touches | ceiling 21.78 times | 43.64 ↔ 206,388 | The third column is the rule of the course: **a display is as correct as the number the reader can read back from it; a display that cannot be read back counts as unreported.** In the first eight rows what is read back is an m³ value, a number of peaks, or a number of series, and the difference column gives what the channel lost. In the middle five rows the reader is a decision maker, and what is read back is a decision, not a single number: how many criteria are seen, where a criterion can be moved without moving what it measures, whether the same report can be produced a second time. In the last five rows the number read back comes not from a display but from **the scale**: the records moved, the coordination overhead item, the values read, the speedup in times. The rule does not change. The divisibility of a computation is known only as far as it is counted, and the item that is not counted turns up not inside the computation but inside the bill. The six courses of the Data Analytics curriculum read the same axis in six different places. | Course | Lessons | Measure axis | Unit | |---|---|---|---| | Introduction to Data Analytics | 9 | the number moves at the moment of definition | definition | | Analysis with Spreadsheets | 11 | where the decision sits | recoverability of the decision | | Data Collection and Preparation | 17 | the row count changes on the way | rows in and out | | Data Analysis with Python | 22 | what the abstraction gains and hides | gain and hidden decision | | Statistical Analysis and Exploratory Data Analysis | 13 | the margin of variation of the number | share | | Visualization, Reporting and Analysis at Scale | 18 | the number the channel makes you read back | read-back error | | Total | 90 | | | Ninety lessons established a single rule six times: **if the decision that produced a number is not written next to it, that number is not a finding but a choice.** It was tested in a different place in each course — in the definition, the cell of the spreadsheet, the preparation step, the default of the call, the margin of variation, and the encoding channel — and each time it was paid in the same form: from the same data, to the same question, two defensible decisions gave two numbers, both printed side by side. The 4.463 m³ difference between the east and the north stood in all of these ninety lessons and said nothing by itself anywhere. Ninety lessons measured every step from data to decision and showed at each that the decision moves the number. But all of these decisions were made by a **human**: which rows will be counted, which threshold will be chosen, which channel will be used, by which key the job will be divided. Which number a procedure that **learns** the decision from the data moves, and where the decisions of that procedure itself stand, was never asked. The next curriculum, Machine Learning and Data Science, opens with this question.