Distribution Visualization

The previous lesson reduced every region to a single number, and the channel carried that single number. Some of the questions in the report cannot be answered by a single number: are the 921 readings of north gathered around a single value or are there two separate groups of subscribers, and do the largest readings stand in the display. These questions look at a distribution, and a distribution does not fit in a single position.

In the previous course this distribution had been built as numbers: the five-number summary was computed and a second set giving the same five numbers was produced. That computation is not repeated; it is taken as input. The question of this lesson is a step further: when the same distribution is encoded with three separate displays, which distributional feature can the reader read back.

• CD8. A distribution display is also a channel. Bin counts put the counts on the length channel; the box summary and the density display put the values on the position channel. The resolutions of the previous lesson continue as they are: length 40, position 100 steps.
• CD9. What is read back is a distributional feature. Three features are counted: the number of peaks, the skew direction, and whether the extreme value is visible.
• CD10. The peak rule is written. A point is a peak if it is larger than both of the first different values on its two sides and is not below 0.3 times the highest value. The rule is the same in all three displays and is applied to the read-back values.
• CD11. The bin width and the kernel width are decisions. Both change not the source but the shape that is read back.
• CD12. The visibility of the extreme value does not depend on the extreme value. The bars are scaled to the highest count; the same extreme value disappears in a display whose neighbor is higher.
• CD13. The box summary carrying no peak is not a flaw but its scope. Five numbers report five points of the ordering; how the values are laid out between those points never enters the box.

The Shape in the Source

# display.py — MODEL: the measurement network of K03-K05 (a fictional setup) is
# generated with the same seed. The 921 readings of north and the twin set of
# K05 are the input; neither is recomputed here, both are passed through the
# display channel.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
PERIODS = ["2026-01", "2026-02", "2026-03"]
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_kept, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (_dropped if r() < 0.046 else _kept).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {a["subscriber_id"] for a in _kept}

READINGS = []
for a in _kept + _dropped:
    r = rng(SEED + 7000 + a["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100
        if a["subscriber_id"] not in VALID:
            r()
            continue
        READINGS.append({"subscriber_id": a["subscriber_id"],
                         "region": a["region"], "period": PERIODS[d],
                         "consumption_m3": t})
        r()

north = sorted(o["consumption_m3"] for o in READINGS if o["region"] == "north")


def push(chunk, end, p=4):
    a, z = chunk[0], chunk[-1]
    u = [(x - a) / (z - a) for x in chunk]
    return [z - (z - a) * (1 - t) ** p if end == "high" else a + (z - a) * t ** p
            for t in u]


CUTS = [0, 230, 460, 690, len(north) - 1]
twin = (push(north[CUTS[0]:CUTS[1] + 1], "high")[:-1]
        + push(north[CUTS[1]:CUTS[2] + 1], "low")[:-1]
        + push(north[CUTS[2]:CUTS[3] + 1], "high")[:-1]
        + push(north[CUTS[3]:], "low"))

# Channels: the bin-count bar is length; the box and the density are position
# on a shared axis.
RES = {"length": 40, "position": 100}


def snap(t, n):
    return round(t * n) / n


def decode_length(v, top):
    return snap(v / top, RES["length"]) * top if top > 0 else 0.0


def decode_position(v, low, top):
    return low + (top - low) * snap((v - low) / (top - low), RES["position"])


def percentile(v, p):
    s = sorted(v)
    i = p * (len(s) - 1)
    a = int(i)
    b = min(a + 1, len(s) - 1)
    return s[a] + (s[b] - s[a]) * (i - a)


# Shape in the source: the number of peaks is known from the setup itself (north
# was generated from a single base, the twin was pushed to two ends). The skew
# direction is the sign of mean minus median; the threshold is 0.05 m3, because
# a difference smaller than this does not reach the text in a number written
# with two decimals. Extreme value: the zone in which the largest observation
# falls.
THRESHOLD_SKEW = 0.05
SETS = [("north", north, 1), ("twin", twin, 2)]


def direction(d):
    return "right" if d > THRESHOLD_SKEW else ("left" if d < -THRESHOLD_SKEW else "symmetric")


SOURCE = {}
for name, v, peaks in SETS:
    mean, med = sum(v) / len(v), percentile(v, 0.50)
    SOURCE[name] = (peaks, direction(mean - med), max(v))
    print(f"{name:6s} n={len(v)} mean {mean:.3f} median {med:.3f} "
          f"largest {max(v):.2f} m3")
    print(f"       shape in the source: peaks {peaks}, skew {direction(mean - med)}, "
          f"extreme value {max(v):.2f} m3")

north  n=921 mean 20.210 median 19.660 largest 45.47 m3
       shape in the source: peaks 1, skew right, extreme value 45.47 m3
twin   n=921 mean 19.754 median 19.660 largest 45.47 m3
       shape in the source: peaks 2, skew right, extreme value 45.47 m3

The number of peaks in the source is not estimated, it is known: both sets are fictional, and how they were built is in the code. The 921 readings of north were generated from a single base value; the twin set was obtained by pushing the sorted values of north to two ends in quarter slices — one has one gathering center, the other has two.

The skew direction is read from the sign of mean minus median; the threshold was written as 0.05 m³, because a smaller difference does not make it into the text in a number reported with two decimals. In north the difference is 0.550 m³, in the twin 0.094 m³, and the second is just above the threshold.

The Same Distribution Passes Through Three Displays

# three_displays.py — the same distribution is encoded with three displays and
# three distributional features are tried to be read back. The peak rule is the
# same in every display: a point larger than the different values on its two
# sides and not below 0.3 times the highest.
WIDTH, AXIS, N = 4.0, 48.0, 12
H, M = 2.0, 60


def bin_counts(v, width, n):
    c = [0] * n
    for x in v:
        c[min(int(x / width), n - 1)] += 1
    return c


def peaks(y, ratio=0.3):
    m, t = max(y), []
    for i in range(len(y)):
        if i and y[i] == y[i - 1]:
            continue
        before = next((y[j] for j in range(i - 1, -1, -1) if y[j] != y[i]), None)
        after = next((y[j] for j in range(i + 1, len(y)) if y[j] != y[i]), None)
        if before is not None and after is not None and y[i] > before and y[i] > after and y[i] >= ratio * m:
            t.append(i)
    return t


def summarize(x, w):
    tot = sum(w)
    mean, cum = sum(a * b for a, b in zip(x, w)) / tot, 0.0
    for a, b in zip(x, w):
        cum += b
        if cum >= tot / 2:
            return mean, a
    return mean, x[-1]


def display_bins(v, width=WIDTH):
    n = round(AXIS / width)
    c = bin_counts(v, width, n)
    g = [round(decode_length(x, max(c))) for x in c]
    mean, med = summarize([(i + 0.5) * width for i in range(n)], g)
    return len(peaks(g)), direction(mean - med), g[min(int(max(v) / width), n - 1)] > 0, g


def display_box(v):
    b = [min(v), percentile(v, 0.25), percentile(v, 0.50), percentile(v, 0.75), max(v)]
    g = [decode_position(x, 0.0, AXIS) for x in b]
    iqr = g[3] - g[1]
    low_fence, high_fence = g[1] - 1.5 * iqr, g[3] + 1.5 * iqr
    outside = [x for x in v if x < low_fence or x > high_fence]
    dots = len({decode_position(x, 0.0, AXIS) for x in outside})
    return None, direction((g[3] - g[2]) - (g[2] - g[1])), dots > 0, (len(outside), dots)


def display_density(v, h=H):
    x = [AXIS * i / (M - 1) for i in range(M)]
    y = [sum(math.exp(-0.5 * ((a - t) / h) ** 2) for t in v) for a in x]
    g = [decode_position(t, 0.0, max(y)) for t in y]
    mean, med = summarize(x, g)
    return (len(peaks(g)), direction(mean - med),
            g[round((M - 1) * max(v) / AXIS)] > 0, g)


DISPLAYS = [("bin counts", display_bins), ("box summary", display_box),
            ("density", display_density)]
print(f"bin width {WIDTH:.0f} m3, kernel width {H:.0f} m3, "
      f"axis 0-{AXIS:.0f} m3, {M} sampling points")
_c = bin_counts(twin, WIDTH, N)
_g = [round(decode_length(x, max(_c))) for x in _c]
print("\ntwin set, when the counts pass through the length channel (source / read back):")
for i in range(N):
    print(f"{i * WIDTH:5.0f}-{(i + 1) * WIDTH:<3.0f}{_c[i]:5d}{_g[i]:6d} "
          f"|{'#' * round(40 * _g[i] / max(_g))}")
_lost = [(i, _c[i]) for i in range(N) if _c[i] > 0 and _g[i] == 0]
print(f"bins dropped to zero on read-back: {len(_lost)}, "
      f"readings inside them: {sum(x for _, x in _lost)}\n")
print(f"{'set':7s}{'display':16s}{'peaks':>12s}{'skew':>12s}{'extreme value':>14s}"
      f"{'features passed':>17s}")
for name, v, _ in SETS:
    kp, kd, _ke = SOURCE[name]
    for d_name, f in DISPLAYS:
        t, c, u, _ex = f(v)
        passed = sum([t == kp, c == kd, u])
        print(f"{name:7s}{d_name:16s}{str(t) if t is not None else 'unreadable':>12s}"
              f"{c:>12s}{'yes' if u else 'no':>14s}{passed:>14d}/3")

bin width 4 m3, kernel width 2 m3, axis 0-48 m3, 60 sampling points

twin set, when the counts pass through the length channel (source / read back):
    0-4     32    33 |####
    4-8      0     0 |
    8-12     0     0 |
   12-16   318   321 |#######################################
   16-20   114   115 |##############
   20-24   110   107 |#############
   24-28   329   329 |########################################
   28-32    11     8 |#
   32-36     4     0 |
   36-40     0     0 |
   40-44     2     0 |
   44-48     1     0 |
bins dropped to zero on read-back: 3, readings inside them: 7

set    display                peaks        skew extreme value  features passed
north  bin counts                 1       right           yes             3/3
north  box summary       unreadable       right           yes             2/3
north  density                    1       right           yes             3/3
twin   bin counts                 2       right            no             2/3
twin   box summary       unreadable       right           yes             2/3
twin   density                    2        left            no             1/3

The bar table shows directly what the channel does. A bar of forty steps is scaled to the highest count; in the twin the highest count is 329, so one step corresponds to 8.2 readings. Every bin with fewer than five readings stays below half a step and is read back as zero. Three bins are erased this way, and the 7 readings inside them leave the display entirely; the 45.47 m³ extreme value is among them.

The same extreme value is read back in north, and the only reason is CD12: in north the highest count is 234, one step is 5.9 readings, and the three readings in the 44–48 m³ bin are enough to fill a step. The extreme value is in the same place and at the same count in both sets; what determines whether it appears is the height of the tallest bar.

In the feature table no display passes all three, and each one captures something the others miss. The box summary carries no peak count at all — the column says “unreadable”, because there is no way to derive a peak from five numbers. In return it is the only display that has the extreme value read back in both sets: the observations outside the fence are printed as separate dots, they are not scaled, and they do not look at their neighbors. Bin counts and density are the only displays that read back the two peaks of the twin, and both lose the extreme value, because both are scaled to the highest value.

That the density display reads the skew of the twin as reversed is not a display flaw but the result of a source that sits close to the threshold: an asymmetry of 0.094 m³ cannot keep its direction when it is reduced to sixty sampling points and a position grid of a hundred steps. For a distributional feature this close to the decision threshold, the display choice determines the result, and that feature is not reported.

The Bin Width Is a Decision

# decision.py — the bin width and the kernel width are decisions; both move the
# number of peaks read back. The number of peaks in the source does not change.
print(f"{'width':>9s}{'bins':>8s}" + "".join(f"{name:>9s}" for name, _, _ in SETS))
for width in (2.0, 3.0, 4.0, 6.0, 8.0, 12.0):
    print(f"{width:8.0f} m3{round(AXIS / width):8d}"
          + "".join(f"{display_bins(v, width)[0]:9d}" for _, v, _ in SETS))
print(f"{'kernel':>9s}{'samples':>8s}" + "".join(f"{name:>9s}" for name, _, _ in SETS))
for h in (0.5, 1.0, 2.0, 3.0, 5.0, 8.0):
    print(f"{h:8.1f} m3{M:8d}"
          + "".join(f"{display_density(v, h)[0]:9d}" for _, v, _ in SETS))
print("peaks in the source: " + ", ".join(f"{name} {t}" for name, _, t in SETS))

    width    bins    north     twin
       2 m3      24        1        2
       3 m3      16        1        2
       4 m3      12        1        2
       6 m3       8        1        2
       8 m3       6        1        2
      12 m3       4        1        1
   kernel samples    north     twin
     0.5 m3      60        4        2
     1.0 m3      60        1        2
     2.0 m3      60        1        2
     3.0 m3      60        1        2
     5.0 m3      60        1        1
     8.0 m3      60        1        1
peaks in the source: north 1, twin 2

The number of peaks in the source does not change in either row: one in north, two in the twin. The only thing that changes is the number read back, and the only thing that changes it is a decision.

With the bin width between 2 and 8 m³, the two peaks of the twin are read back. At 12 m³ the number falls to one, because the distance between the two gathering centers is about 12 m³ and a single bin swallows both. The previous course had pointed at this trap with a single sentence; here the place of the threshold is counted.

The kernel width goes wrong in the opposite direction as well. At 0.5 m³, four peaks are read back in north, although there is one in the source. At this width the kernel is narrower than the 0.81 m³ gap between sampling points; instead of smoothing the data the curve follows individual readings, and random fluctuations in the reading density are read back as peaks. At 5 m³ and above, the two peaks of the twin merge. Between the two decisions there is not a preference but two separate errors: a narrow choice produces the peak that is not there, a wide choice erases the one that is. The rule is the concrete form of CD11 — both widths are written under the output, otherwise the number of peaks read back looks like a finding.

Summary

• Bin counts put the counts on the length channel, while the box summary and the density display put the values on the position channel; what is read back is not an m³ value but the number of peaks, the skew direction, and the extreme value.
• No display passes all three features: the box summary carries no peak at all but is the only display that has the extreme value read back in both sets; bin counts and density read back the two peaks but lose the extreme value.
• In the counts that pass through the length channel three bins drop to zero and 7 readings are erased; the same 45.47 m³ extreme value shows in north because the tallest bar there is 234, not 329.
• A bin width of 12 m³ lowers the number of peaks read back in the twin from two to one, and a kernel width of 0.5 m³ shows four peaks in north that are not there; the number of peaks in the source does not change.
• The skew of 0.094 m³ in the twin is just above the threshold and its direction changes with the display; a distributional feature this close to the threshold is not reported.

Next Step

In this lesson there was a continuous value on one side of the axis, and the display turned it into bins or a curve. Most of the questions in the report, however, are categorical: the number of subscribers in the four tariff tiers of five regions and the share of those numbers within the region. Categorical display has a decision of its own — whether the slices will be stacked or placed side by side — and this is a channel problem: in a stacked display only the bottom slice has a baseline. The next lesson counts the read-back error of the upper slices by slice order and compares it with the grouped display.