Relationship Visualization

The first lesson, while binding the four question types to channels, had set the relationship question apart: this question asks for two values to be read back at the same time, and for that a second independent copy of the channel is needed. Only position has two copies, the horizontal and the vertical axis. This lesson uses those two copies.

The pair used is the natural question of the measurement network: how well a subscriber’s first period reading reports the subscriber’s general level. The strength of the relationship was measured with a coefficient in the previous course, and here it is neither computed nor interpreted. The question here belongs to display: when the same relationship is encoded in three forms, what reaches the reader, and what is lost as the number of points grows.

• CD20. The relationship question asks for two position channels at the same time. It is the result of the first lesson and is used here as input.
• CD21. The drawn point is a mark, and it has a width. The value of a single point is read from the position channel with 100 steps, but two points are distinguished only when they are apart by more than the mark width. The mark width was taken as one fortieth of the axis.
• CD22. An overlapping point is not read back. The reader sees a cell once; the measure is how many points remain in a separate cell.
• CD23. The heat map does not lose points, it loses counts. Every point is counted in a cell, but the count descends to a color step, and the number of levels that can be read back is limited by the number of steps.
• CD24. The color step is divided not by the distribution of the counts but by the boundary that is set. If the counts are skewed, in equal-interval steps the lower steps fill up and the upper steps stay empty.
• CD25. A display that gives a single y for an x does not carry the span at that x. The number the line has read back is one step; the number of steps the cloud covers is a separate measure.

Two Position Channels and the Mark Width

# relationship.py — MODEL: the measurement network of K03-K05 is
# generated with the same seed. The relationship pair: the subscriber's first
# period reading and the same subscriber's mean.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
PERIODS = ["2026-01", "2026-02", "2026-03"]
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_kept, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (_dropped if r() < 0.046 else _kept).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {a["subscriber_id"] for a in _kept}

READINGS = []
for a in _kept + _dropped:
    r = rng(SEED + 7000 + a["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            a["base"] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100
        if a["subscriber_id"] not in VALID:
            r()
            continue
        READINGS.append({"subscriber_id": a["subscriber_id"],
                         "region": a["region"], "period": PERIODS[d],
                         "consumption_m3": t})
        r()

from collections import defaultdict

# The first is the subscriber's first period reading, the second is the mean
# of all the periods of the same subscriber. Subscribers with at least two
# readings are taken.
_by_subscriber = defaultdict(list)
for o in READINGS:
    _by_subscriber[o["subscriber_id"]].append((o["period"], o["consumption_m3"]))
PAIRS = []
for _id in sorted(_by_subscriber):
    v = sorted(_by_subscriber[_id])
    if len(v) >= 2:
        PAIRS.append((v[0][1], sum(x for _, x in v) / len(v)))

# Two points are distinguished only when they are apart by more than the mark
# width. The drawn point is a mark and it has a width; the mark width was taken
# as one fortieth of the axis. The value of a single point, on the other hand,
# is read from the position channel, with 100 steps.
RES_POSITION, MARK = 100, 40


def round_ten(x):
    return (int(x) // 10 + 1) * 10


AXIS_X = round_ten(max(a for a, _ in PAIRS))
AXIS_Y = round_ten(max(b for _, b in PAIRS))


def cell(p):
    return (min(MARK - 1, int(MARK * p[0] / AXIS_X)),
            min(MARK - 1, int(MARK * p[1] / AXIS_Y)))


# The choice of the subset comes from the seed: the pair list is shuffled once
# and then taken from the start. This way every size is a prefix of the same
# order.
_r = rng(SEED + 404)
ORDER = list(range(len(PAIRS)))
for i in range(len(ORDER) - 1, 0, -1):
    j = int(_r() * (i + 1))
    ORDER[i], ORDER[j] = ORDER[j], ORDER[i]
SHUFFLED = [PAIRS[i] for i in ORDER]

print(f"fictional relationship set: {len(PAIRS)} subscriber pairs "
      f"(first period reading, subscriber mean)")
print(f"axes 0-{AXIS_X} and 0-{AXIS_Y} m3; position resolution {RES_POSITION} "
      f"steps, mark width 1/{MARK} of the axis")
print(f"plane {MARK}x{MARK} = {MARK * MARK} distinguishable cells")

fictional relationship set: 1101 subscriber pairs (first period reading, subscriber mean)
axes 0-60 and 0-50 m3; position resolution 100 steps, mark width 1/40 of the axis
plane 40x40 = 1600 distinguishable cells

Two resolution numbers stand side by side, and they are not the same thing. The value of a single point is read from the position channel and there are 100 steps on the axis; it is the number of the first lesson. But for two points to be seen as two separate points, the distance between them has to exceed the width of the mark itself. The drawn point is not a mathematical point; it takes up a place on the page, and that place was taken as one fortieth of the axis. For this reason the plane is divided not into 100 times 100 but into 40 times 40 distinguishable cells, and 1,101 points will be put on 1,600 cells.

As the Number of Points Grows

# scatter.py — as the number of points grows, how many points remain in a
# separate cell. An overlapping point is not read back: the reader sees the
# cell once.
print(f"{'points':>7s}{'filled cells':>14s}{'hidden':>10s}{'hidden %':>12s}"
      f"{'densest cell':>16s}")
for n in (100, 250, 500, 800, len(SHUFFLED)):
    s = defaultdict(int)
    for p in SHUFFLED[:n]:
        s[cell(p)] += 1
    print(f"{n:7d}{len(s):14d}{n - len(s):10d}{100 * (n - len(s)) / n:11.1f}%"
          f"{max(s.values()):16d}")

 points  filled cells    hidden    hidden %    densest cell
    100            85        15       15.0%               3
    250           161        89       35.6%               5
    500           232       268       53.6%               9
    800           266       534       66.8%              12
   1101           309       792       71.9%              16

There are 1,600 cells in the plane and 1,101 points. There seems to be plenty of room, but the 1,101 points fall into only 309 cells, and 792 points, that is 71.9%, have another point on top of them. The reason is that the points are not distributed evenly over the plane: where the relationship is, the points overplot, and precisely the region that carries the most information becomes the region that is least readable. In the densest cell 16 points overplot, and for the reader that cell is no different from a single point.

How the column grows is also worth recording. At a hundred points the hidden rate is 15.0%; at two hundred and fifty points 35.6%; at five hundred 53.6%. While the number of points rises elevenfold, the number of filled cells rises only from 85 to 309, that is, threefold and a half. The information a scatter plot carries grows not with the number of points but with the number of filled cells, and this number has a ceiling: 1,600.

The Heat Map and the Step Boundary Decision

# heatmap.py — the same pair set is divided into coarse intervals, the count
# of every cell is turned into a color step and read back. The color is not a
# name but a step number; the number of steps is 7, as in the first lesson.
NX, NY, STEPS = 12, 10, 7
GX, GY = AXIS_X / NX, AXIS_Y / NY
COUNT = defaultdict(int)
for a, b in PAIRS:
    COUNT[(min(NX - 1, int(a / GX)), min(NY - 1, int(b / GY)))] += 1
ENC = max(COUNT.values())


def encode_color(c):
    return min(STEPS - 1, int(STEPS * c / ENC))


def decode_color(b):
    return (b + 0.5) * ENC / STEPS


print(f"{NX}x{NY} intervals ({GX:.0f}x{GY:.0f} m3), densest cell {ENC} points, "
      f"{STEPS} color steps")
for j in range(NY - 1, -1, -1):
    print(f"{(j + 1) * GY:5.0f} |" + "".join(
        str(encode_color(COUNT[(i, j)])) if COUNT[(i, j)] else "." for i in range(NX)))
print("      +" + "-" * NX + f"  x axis 0-{AXIS_X} m3")

filled = [c for c in COUNT.values() if c > 0]
levels_src = len(set(filled))
levels_read = len({encode_color(c) for c in filled})
rel = sum(abs(decode_color(encode_color(c)) - c) / c for c in filled) / len(filled)
print(f"filled cells {len(filled)}, no point was lost: "
      f"{sum(filled)} = {len(PAIRS)}")
print(f"the source has {levels_src} distinct count values, levels read back {levels_read}"
      f"/{STEPS}; mean relative read-back error {100 * rel:.1f}%")
print("step occupancy: " + ", ".join(
    f"{b}:{sum(1 for c in filled if encode_color(c) == b)}" for b in range(STEPS)))

# The second decision: the steps are divided not into equal intervals but into
# an equal number of cells. The boundaries come from the sorted counts, and the
# representative value of each step is the median of its own cells.
_s = sorted(filled)
BOUNDS = [_s[round(len(_s) * k / STEPS)] for k in range(1, STEPS)]


def encode_rank(c):
    b = 0
    while b < STEPS - 1 and c >= BOUNDS[b]:
        b += 1
    return b


REPRESENTATIVE = []
for b in range(STEPS):
    g = sorted(c for c in filled if encode_rank(c) == b)
    REPRESENTATIVE.append(g[len(g) // 2] if g else 0)
rel2 = sum(abs(REPRESENTATIVE[encode_rank(c)] - c) / c for c in filled) / len(filled)
print(f"equal cell count decision: levels read back "
      f"{len({encode_rank(c) for c in filled})}/{STEPS}, "
      f"mean relative error {100 * rel2:.1f}%")
print("step occupancy: " + ", ".join(
    f"{b}:{sum(1 for c in filled if encode_rank(c) == b)}" for b in range(STEPS)))

12x10 intervals (5x5 m3), densest cell 140 points, 7 color steps
   50 |............
   45 |........0...
   40 |...000.000..
   35 |...00000000.
   30 |..00121000..
   25 |..044310....
   20 |004641000...
   15 |0142000.....
   10 |00000.......
    5 |.0..........
      +------------  x axis 0-60 m3
filled cells 51, no point was lost: 1101 = 1101
the source has 28 distinct count values, levels read back 6/7; mean relative read-back error 280.9%
step occupancy: 0:37, 1:5, 2:2, 3:1, 4:5, 5:0, 6:1
equal cell count decision: levels read back 6/7, mean relative error 15.7%
step occupancy: 0:0, 1:13, 2:7, 3:8, 4:8, 5:8, 6:7

The heat map does not lose what the scatter plot loses: all 1,101 points were counted in a cell and the total matches exactly. What it loses is something else — the count itself. There are 28 distinct count values in the source; the color step reduces them to seven, and in fact to six.

The step occupancies are the direct evidence of CD24. The equal-interval division takes the 140 points of the densest cell as the measure and piles 37 of the 51 filled cells onto a single step, the lowest. The middle value of that step is 10 points; there are also single-point cells inside it. For this reason the mean relative read-back error comes out at 280.9%, and the reader sees most of the plane as a single level — the zeros lined up next to the zeros in the plane show this.

The second decision encodes the same data with the same seven steps, only putting the boundaries somewhere else: the steps are divided not into equal intervals but into an equal number of cells. The occupancy is balanced and the mean relative error falls to 15.7%, to one eighteenth. It has a price, and it has to be written: a step series divided into an equal number of cells carries not the magnitude but the order. With the first decision the step difference of two cells tells how many points there are; with the second, only which one is more crowded. The same data, the same display, two defensible decisions, and an eighteenfold difference.

What the Line Does Not Carry

# line.py — the third display: a line that gives a single y for every x. The
# coefficient of the line was computed in M26/K05, and here it is neither
# computed nor interpreted; the only thing measured is how much of the y at
# that x a single y carries. The span is written in steps of the position
# channel (the axis is divided into 100 steps).
print(f"{'x interval':>12s}{'points':>7s}{'y smallest':>12s}{'y largest':>12s}"
      f"{'span steps':>17s}")
steps_list = []
for i in range(NX):
    ys = [b for a, b in PAIRS if min(NX - 1, int(a / GX)) == i]
    if len(ys) < 20:
        continue
    n = round(RES_POSITION * (max(ys) - min(ys)) / AXIS_Y)
    steps_list.append(n)
    print(f"{i * GX:6.0f}-{(i + 1) * GX:<5.0f}{len(ys):7d}{min(ys):12.2f}"
          f"{max(ys):12.2f}{n:17d}")
print(f"the line gives 1 step for every x; the cloud covers on average "
      f"{sum(steps_list) / len(steps_list):.1f} steps")

  x interval points  y smallest   y largest       span steps
     0-5         36        5.82       19.04               26
     5-10        30        3.49       19.43               32
    10-15       205        5.46       26.61               42
    15-20       295        6.57       35.66               58
    20-25       235        7.84       38.13               61
    25-30       154       12.01       35.71               47
    30-35        84       13.91       33.52               39
    35-40        34       17.53       38.84               43
    40-45        20       18.87       43.48               49
the line gives 1 step for every x; the cloud covers on average 44.1 steps

The third display is the one that takes up the least space and throws away the most. A line gives a single y for every x; that value is a single step of the position channel. In the same x interval the observations cover on average 44.1 steps, that is, the number the line has read back takes up about a forty-fourth of the vertical span of the cloud. In the 15–20 m³ interval there are 295 subscribers and their means vary between 6.57 and 35.66 m³; the line puts a single number on this interval.

This is not an objection to the line. The line exists in the first place to throw the span away, and if the report asks for a trend it is the right display. What is to be recorded is that what is thrown away is countable: when the span is not written next to the line, the reader takes a cloud of forty-four steps for one step.

Summary

• The relationship is encoded with two position channels; the value of a single point is read back with 100 steps, but the separation of two points depends on the mark width, and the plane comes down to 40 times 40 cells.
• 1,101 points fall into only 309 cells: 792 points, 71.9%, overlap, and 16 points stand in the densest cell. The information a scatter plot carries grows not with the number of points but with the number of filled cells.
• The heat map loses no point but loses the count: the 28 distinct count values in the source fall to six of the seven steps, 37 of the 51 filled cells land on a single step, and the relative error is 280.9%.
• When the step boundary is set at an equal number of cells instead of equal intervals, the same error falls to 15.7%; the price is that the step now carries not the magnitude but the order.
• A line that gives a single y for every x reduces a span of 44.1 steps on average to a single step; when the span is not written, the reader takes the cloud for a line.

Next Step

In four lessons four channels were measured: position, length, area, and color step — all on a single snapshot. The mean of five regions was at a moment, the distribution of north was the total of a period, the crosstab was a cross section, and both axes of the scatter plot belonged to the same time. When one of the axes is time, what the channel carries changes: the reader now reads not a value but a change, and the change itself is an unmeasured number. The next lesson moves to that axis and measures the read-back of the slope — what happens to the slope read back when the same series is encoded with two axis scales, and whether a trend can be read without separating the seasonal component.