Statistical Programming Languages

The previous topic solved the scale problem by changing tools: the column was read, processed piece by piece, and the computation was taken to the data. In every step, what changed was where the tool was placed. The only thing that stayed the same was the language itself; the analysis was written from start to finish in the same style, with the same statements, and it was never asked whether this style was a choice.

This topic asks that question. The same analysis is expressed in other environments in other styles, and the difference between them is not speed but the length of the expression. This is the first number of this lesson: how many statements, how many intermediate variables and how many loops the same analysis is written with. Beside it stands the second number of the course: as the expression gets shorter, how many decisions move out of the code and into a default.

• AE1. The dataset is inherited from Data Collection and Preparation and is fiction: a long-format reading table of 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code and the numbers are the same from run to run.
• AE2. In this lesson no language, library or environment name appears. Alternative environments are referred to by their approaches, and the approach is modeled in Python. The purpose of the model is not to imitate a tool but to make the expression style measurable.
• AE3. The three approaches are as follows. In the procedural expression the unit of work is the row: the data is walked through one element at a time. The whole-object expression is the approach of analysis-oriented languages and its unit of work is the entire column: no loop is written, the operation is applied to the whole column. In the declarative expression the definition of the result is written, and the order of execution is not.
• AE4. The measure is not a comparative superiority. No ranking is built that says a short expression is good and a long expression is bad. The only things counted are statements, intermediate variables, loops and defaults.

The Same Analysis, Three Expressions

The analysis is defined in a single sentence: mean consumption per region, the top three regions in descending order. The dataset is the input of all three expressions and the seed is fixed.

# environment.py — MODEL: the 3,199-row fiction dataset of Data Collection and
# Preparation is regenerated from the same seed; no tool, library or
# environment is imitated.
import ast
import inspect
import math

SEED = 20260218
RAW = 1400
PERIODS = ["2026-01", "2026-02", "2026-03"]
TARIFF = [10, 25, 40]
M32 = 0xFFFFFFFF


def mix(x: int) -> int:
    x = ((x ^ (x >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    return (x ^ (x >> 16)) & M32


def rng(t: int):
    s = [mix(t)]

    def next_value() -> float:
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u: float, w: list[float]) -> int:
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]
_subscribers, _dropped = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    record = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2],
              "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
              "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1}
    (_dropped if r() < 0.046 else _subscribers).append(record)
_valid = {a["subscriber_id"] for a in _subscribers}
READINGS = []
for k in _subscribers + _dropped:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100
            + 0.5) / 100
        if k["subscriber_id"] not in _valid:
            r()
            continue
        READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"],
                         "period": PERIODS[d], "consumption_m3": t,
                         "satisfaction": k["satisfaction"],
                         "tier": 1 if t < TARIFF[0] else 2 if t < TARIFF[1]
                         else 3 if t < TARIFF[2] else 4})
        r()

print(f"fictional dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers, "
      f"{sum(o['consumption_m3'] for o in READINGS):.1f} m3")
print(f"unread rows (zero written): {sum(1 for o in READINGS if o['consumption_m3'] == 0)}")

fictional dataset (seed 20260218): 3199 rows, 1329 subscribers, 63060.6 m3
unread rows (zero written): 119

All three expressions take the same dataset. The measurement is made not by hand but from the code’s own tree: the source of each expression is parsed, and the statements in its body, the names assigned and the loop nodes are counted. This way the answer to the question “how many steps” is not a comment but a countable quantity.

# Helpers of the whole-object approach: the operation is applied to the entire column.
def column(rows, name):
    return [r[name] for r in rows]


def group_mean(keys, measure):
    total, count = {}, {}
    for a, v in zip(keys, measure):
        total[a] = total.get(a, 0.0) + v
        count[a] = count.get(a, 0) + 1
    return {a: total[a] / count[a] for a in total}


def descending(mapping):
    return sorted(mapping.items(), key=lambda p: -p[1])


def top_n(pairs, n):
    return pairs[:n]


# Solver of the declarative approach: the request is a definition, the order of execution is here.
def solve(request):
    keys = column(request["source"], request["group"])
    values = column(request["source"], request["measure"])
    return top_n(descending(group_mean(keys, values)), request["top"])


HELPERS = {"column": column, "group_mean": group_mean, "descending": descending,
           "top_n": top_n, "solve": solve}


def procedural(rows):
    total, count = {}, {}
    for row in rows:
        region = row["region"]
        total[region] = total.get(region, 0.0) + row["consumption_m3"]
        count[region] = count.get(region, 0) + 1
    mean = {}
    for region in total:
        mean[region] = total[region] / count[region]
    ranked = sorted(mean.items(), key=lambda p: -p[1])
    return ranked[:3]


def whole_object(rows):
    region = column(rows, "region")
    consumption = column(rows, "consumption_m3")
    mean = group_mean(region, consumption)
    return top_n(descending(mean), 3)


def declarative(rows):
    return solve({"source": rows, "group": "region", "measure": "consumption_m3", "top": 3})


# The measurement is read from the code's tree: statement count, assigned name count, loop nodes.
LOOPS = (ast.For, ast.While, ast.comprehension)


def tree(f):
    return ast.parse(inspect.getsource(f)).body[0]


def statement_count(f):
    return len(tree(f).body)


def name_count(f):
    g = tree(f)
    return len({n.id for n in ast.walk(g)
                if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)})


def written_loops(f):
    return sum(1 for n in ast.walk(tree(f)) if isinstance(n, LOOPS))


# Executed loops: the loops of the helpers the expression calls are added to its own loops.
def executed_loops(f, seen=None):
    seen = set() if seen is None else seen
    count = written_loops(f)
    for n in ast.walk(tree(f)):
        if isinstance(n, ast.Call) and isinstance(n.func, ast.Name):
            name = n.func.id
            if name in HELPERS and name not in seen:
                seen.add(name)
                count += executed_loops(HELPERS[name], seen)
    return count


def align(w):
    return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j])
                             for j, v in enumerate(h))


EXPRESSIONS = [("procedural", procedural), ("whole-object", whole_object),
               ("declarative", declarative)]
y = align([16, 11, 23, 16, 15])
print("\n" + y(["expression", "statement", "intermediate variable", "written loop",
                "executed loop"]))
for name, f in EXPRESSIONS:
    print(y([name, statement_count(f), name_count(f), written_loops(f), executed_loops(f)]))

results = [f(READINGS) for _, f in EXPRESSIONS]
same = all(s == results[0] for s in results)
print(f"\nall three give the same result: {'yes' if same else 'no'}")
print("  " + ", ".join(f"{b} {v:.3f}" for b, v in results[0]))

expression        statement  intermediate variable    written loop  executed loop
procedural                6                      6               2              2
whole-object              4                      3               0              3
declarative               1                      0               0              3

all three give the same result: yes
  east 24.674, central 22.396, north 20.210

The last line is the anchor of the lesson: all three expressions give the same three regions and the same three numbers. The difference is not in the result but in the writing. The procedural expression holds 6 statements and binds 6 names; four of them are intermediate results and none of them enters the report. The whole-object expression comes down to 4 statements and 3 names, because rows are no longer walked: region and consumption are taken as two columns and the mean is applied to the whole column. The declarative expression is a single statement and binds no names; what is written is the recipe of the result.

The real change in the second row happens not in the number but in the unit of work. In the procedural expression, what the code talks about is a single row named row, and the total and the count grow out of the fields of that row; the object of the analysis is a record. In the whole-object expression, what the code talks about is two columns named region and consumption; not even the name of a single row appears. The approach of statistical programming languages comes from here: data is handled in the unit of the field, not the record, and since the operation is applied to the whole field, the walking disappears entirely from the visible part of the code.

The Loop Did Not Disappear, It Moved

The last two columns of the table go in opposite directions. Written loops fall from 2 to 0, executed loops rise from 2 to 3. This is one of the most commonly misread places of abstraction: giving up writing the loop does not eliminate the loop, it only changes its place.

The procedural expression writes two loops: one scans the dataset for the total and the count, the other walks the five grouped regions for the division. The whole-object expression writes no loop, but the helpers it calls run three loops — column scans the dataset once each time it is called and it is called twice, group_mean makes one scan and one merge. As the expression gets shorter, the number of passes does not decrease; in this example it increases. Assuming that there is a link between brevity and the number of passes is an unmeasured claim.

What is gained in return is also countable: the four intermediate names in the procedural expression — total, count, mean, ranked — come down to three in the whole-object expression and to zero in the declarative expression. An intermediate name is something that has to be read and can be misused; lowering its count lowers the reading load of the expression. The price is the subject of the next section.

What the measure does not count must also be written. The table counts only the statements the person writing the analysis writes; the bodies of the helpers do not enter the count. The four helpers and the solver together hold nine statements, and these are statements someone wrote somewhere. The gain of abstraction is not that the amount of work decreases but that the same statements are written once and used many times; the two columns are taken with the same column call in three different analyses as well. This is why the measure gives no ranking of superiority; it only shows where the load has moved.

Five Decisions of a Single Statement

The declarative expression writes four things: the source, the grouping field, the measure field and how many rows are requested. These are not all of the decisions that determine the result of the analysis. The solver below meets the same request with a complete set of defaults, and each default is changed one by one.

# The same declarative request, this time in a solver that carries all the defaults by name.
DEFAULTS = {"missing": "count", "weight": "row", "summary": "mean",
            "period": "all", "direction": "descending"}


def median(v):
    s = sorted(v)
    n = len(s)
    return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2


def solve_full(request):
    a = dict(DEFAULTS)
    a.update(request)
    rows = [row for row in request["source"]
            if (a["period"] == "all" or row["period"] == a["period"])
            and (a["missing"] == "count" or row["consumption_m3"] > 0)]
    if a["weight"] == "subscriber":
        unit = {}
        for row in rows:
            unit.setdefault(row["subscriber_id"], (row[a["group"]], []))[1].append(row[a["measure"]])
        rows = [{a["group"]: g, a["measure"]: sum(v) / len(v)} for g, v in unit.values()]
    bucket = {}
    for row in rows:
        bucket.setdefault(row[a["group"]], []).append(row[a["measure"]])
    summary = {g: (sum(v) / len(v) if a["summary"] == "mean" else median(v))
               for g, v in bucket.items()}
    ranked = sorted(summary.items(), key=lambda p: p[1], reverse=a["direction"] == "descending")
    return ranked[:a.get("top", len(ranked))]


BASE = {"source": READINGS, "group": "region", "measure": "consumption_m3", "top": 3}
CHANGES = [("unread rows", "missing", "count", "drop"),
           ("measure weight", "weight", "row", "subscriber"),
           ("summary function", "summary", "mean", "median"),
           ("period scope", "period", "all", "2026-01"),
           ("sort direction", "direction", "descending", "ascending")]
baseline = dict(solve_full(BASE))["east"]
y2 = align([20, 12, 12, 14, 12, 24])
print("\n" + y2(["hidden default", "default", "east m3", "changed", "east m3",
                 "reported top three"]))
values = [baseline]
for name, key, before, after in CHANGES:
    request = {**BASE, key: after}
    top = solve_full(request)
    d = dict(solve_full({k: v for k, v in request.items() if k != "top"}))["east"]
    values.append(d)
    print(y2([name, before, f"{baseline:.3f}", after, f"{d:.3f}",
              " ".join(b for b, _ in top)]))
print(f"\nsingle-statement expression writes {len(BASE)} decisions, leaves {len(DEFAULTS)} "
      f"decisions to defaults")
print(f"if defaults are written the request grows {len(BASE)} -> {len(BASE) + len(DEFAULTS)} "
      f"keys and the expression is still 1 statement")
print(f"east mean moves {max(values) - min(values):.3f} m3 between {min(values):.3f} and "
      f"{max(values):.3f}")

hidden default           default     east m3       changed     east m3      reported top three
unread rows                count      24.674          drop      25.593      east central north
measure weight               row      24.674    subscriber      24.814      east central north
summary function            mean      24.674        median      24.120      east central north
period scope                 all      24.674       2026-01      25.540      east central north
sort direction        descending      24.674     ascending      24.674        west south north

single-statement expression writes 4 decisions, leaves 5 decisions to defaults
if defaults are written the request grows 4 -> 9 keys and the expression is still 1 statement
east mean moves 1.473 m3 between 24.120 and 25.593

A single statement writes four decisions and leaves five decisions to defaults. When the five defaults are changed one by one, the east mean becomes 25.593 — 24.814 — 24.120 — 25.540 m³ respectively instead of 24.674 m³; between the lowest and the highest there is a difference of 1.473 m³, about six percent of the base value. The fifth does not move the number at all but changes the three reported regions completely: in descending order east, central and north are written, in ascending order west, south and north.

The real observation here is this: the same five decisions exist in the procedural expression as well. There too the unread rows are counted, the weight is the row, and the summary is the arithmetic mean. The difference is the number of places where the decisions can be written. In the procedural expression the same five defaults are scattered across three statements — the loop, the division and the sorting lines — and each one sits in a place that can be changed by hand. In the single-statement expression there is no such line; a decision becomes visible only if it is added to the request as a key. When all the defaults are written explicitly, the request goes from four keys to nine keys and the expression is still a single statement — the cost of the record is not the statement but the key.

The rule of the course finds its exact counterpart here. An abstraction shortens the work as much as it hides decisions; a function whose defaults you do not know is a function that decides on your behalf. A short expression is not bad for this reason — not knowing where a decision is written is bad.

Summary

• When the same analysis is written in three expression styles, it gives the same result: east 24.674 — central 22.396 — north 20.210 m³.
• The measure of an expression is the number of statements, intermediate variables and loops: procedural 6/6/2, whole-object 4/3/0, declarative 1/0/0.
• Written loops fall from 2 to 0 while executed loops rise from 2 to 3; abstraction does not remove the loop, it moves it, and in this example it adds one more.
• The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed, the east mean moves within a range 1.473 m³ wide and the sort direction changes the three reported regions completely.
• The same decisions exist in the procedural expression as well; what changes is the number of places where a decision can be written.

Next Step

The declarative expression came down to a single statement, but this descent has a price: a dictionary of four keys does not tell, in order, what an analysis does. The next lesson measures the form in between — the approach that writes the transformation as a chain made of individual verbs. The measure is from the same family: when the same seven-step transformation is written with intermediate variables, how many names are bound; when it is written as nested calls, in how many steps the reading order overlaps with the execution order; when it is written as a chain, which intermediate result’s row count becomes invisible.