--- title: 'Statistical Programming Languages' source: 'https://academia.sh/en/courses/data-analysis-with-python/statistical-programming-languages' course: 'Data Analysis with Python' language: en updated: '2026-10-07T04:31:10+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the procedural, whole-object and declarative expressions of the same analysis by statement, intermediate variable and loop counts' - 'Shows with numbers that abstraction does not remove the loop: written loops decrease while executed loops increase' - 'Counts how many decisions the shortened expression leaves to defaults and computes how far each default moves the reported number' --- # Statistical Programming Languages The same analysis — mean consumption per region and the top three regions in descending order, on the 3,199-row fiction dataset — is written in three expression styles and all three give the same number: the procedural expression holds 6 statements, 6 intermediate variables and 2 loops; the whole-object expression holds 4 statements, 3 intermediate variables and 0 loops; the declarative expression holds 1 statement, 0 intermediate variables and 0 loops. Written loops drop from 2 to 0 while executed loops rise from 2 to 3: the loop does not disappear, it moves into a helper, and one more is added. The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed one by one, the east mean becomes 25.593, 24.814, 24.120 and 25.540 m³ instead of 24.674 m³, and when the sort direction changes, the reported three regions change completely. The previous topic solved the scale problem by changing tools: the column was read, processed piece by piece, and the computation was taken to the data. In every step, what changed was where the tool was placed. The only thing that stayed the same was **the language itself**; the analysis was written from start to finish in the same style, with the same statements, and it was never asked whether this style was a choice. This topic asks that question. The same analysis is expressed in other environments in other styles, and the difference between them is not speed but **the length of the expression**. This is the first number of this lesson: how many statements, how many intermediate variables and how many loops the same analysis is written with. Beside it stands the second number of the course: as the expression gets shorter, how many decisions move out of the code and into a default. - **AE1.** The dataset is inherited from Data Collection and Preparation and is **fiction**: a long-format reading table of 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code and the numbers are the same from run to run. - **AE2.** In this lesson **no language, library or environment name appears**. Alternative environments are referred to by their **approaches**, and the approach is **modeled** in Python. The purpose of the model is not to imitate a tool but to make the expression style measurable. - **AE3.** The three approaches are as follows. In the **procedural expression** the unit of work is the row: the data is walked through one element at a time. The **whole-object expression** is the approach of analysis-oriented languages and its unit of work is the entire column: no loop is written, the operation is applied to the whole column. In the **declarative expression** the definition of the result is written, and the order of execution is not. - **AE4.** The measure is **not a comparative superiority**. No ranking is built that says a short expression is good and a long expression is bad. The only things counted are statements, intermediate variables, loops and defaults. ## The Same Analysis, Three Expressions The analysis is defined in a single sentence: mean consumption per region, the top three regions in descending order. The dataset is the input of all three expressions and the seed is fixed. ```python # environment.py — MODEL: the 3,199-row fiction dataset of Data Collection and # Preparation is regenerated from the same seed; no tool, library or # environment is imitated. import ast import inspect import math SEED = 20260218 RAW = 1400 PERIODS = ["2026-01", "2026-02", "2026-03"] TARIFF = [10, 25, 40] M32 = 0xFFFFFFFF def mix(x: int) -> int: x = ((x ^ (x >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 return (x ^ (x >> 16)) & M32 def rng(t: int): s = [mix(t)] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] record = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(record) _valid = {a["subscriber_id"] for a in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t, "satisfaction": k["satisfaction"], "tier": 1 if t < TARIFF[0] else 2 if t < TARIFF[1] else 3 if t < TARIFF[2] else 4}) r() print(f"fictional dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers, " f"{sum(o['consumption_m3'] for o in READINGS):.1f} m3") print(f"unread rows (zero written): {sum(1 for o in READINGS if o['consumption_m3'] == 0)}") ``` ``` fictional dataset (seed 20260218): 3199 rows, 1329 subscribers, 63060.6 m3 unread rows (zero written): 119 ``` All three expressions take the same dataset. The measurement is made not by hand but from the code's own tree: the source of each expression is parsed, and the statements in its body, the names assigned and the loop nodes are counted. This way the answer to the question "how many steps" is not a comment but a countable quantity. ```python # Helpers of the whole-object approach: the operation is applied to the entire column. def column(rows, name): return [r[name] for r in rows] def group_mean(keys, measure): total, count = {}, {} for a, v in zip(keys, measure): total[a] = total.get(a, 0.0) + v count[a] = count.get(a, 0) + 1 return {a: total[a] / count[a] for a in total} def descending(mapping): return sorted(mapping.items(), key=lambda p: -p[1]) def top_n(pairs, n): return pairs[:n] # Solver of the declarative approach: the request is a definition, the order of execution is here. def solve(request): keys = column(request["source"], request["group"]) values = column(request["source"], request["measure"]) return top_n(descending(group_mean(keys, values)), request["top"]) HELPERS = {"column": column, "group_mean": group_mean, "descending": descending, "top_n": top_n, "solve": solve} def procedural(rows): total, count = {}, {} for row in rows: region = row["region"] total[region] = total.get(region, 0.0) + row["consumption_m3"] count[region] = count.get(region, 0) + 1 mean = {} for region in total: mean[region] = total[region] / count[region] ranked = sorted(mean.items(), key=lambda p: -p[1]) return ranked[:3] def whole_object(rows): region = column(rows, "region") consumption = column(rows, "consumption_m3") mean = group_mean(region, consumption) return top_n(descending(mean), 3) def declarative(rows): return solve({"source": rows, "group": "region", "measure": "consumption_m3", "top": 3}) # The measurement is read from the code's tree: statement count, assigned name count, loop nodes. LOOPS = (ast.For, ast.While, ast.comprehension) def tree(f): return ast.parse(inspect.getsource(f)).body[0] def statement_count(f): return len(tree(f).body) def name_count(f): g = tree(f) return len({n.id for n in ast.walk(g) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)}) def written_loops(f): return sum(1 for n in ast.walk(tree(f)) if isinstance(n, LOOPS)) # Executed loops: the loops of the helpers the expression calls are added to its own loops. def executed_loops(f, seen=None): seen = set() if seen is None else seen count = written_loops(f) for n in ast.walk(tree(f)): if isinstance(n, ast.Call) and isinstance(n.func, ast.Name): name = n.func.id if name in HELPERS and name not in seen: seen.add(name) count += executed_loops(HELPERS[name], seen) return count def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) EXPRESSIONS = [("procedural", procedural), ("whole-object", whole_object), ("declarative", declarative)] y = align([16, 11, 23, 16, 15]) print("\n" + y(["expression", "statement", "intermediate variable", "written loop", "executed loop"])) for name, f in EXPRESSIONS: print(y([name, statement_count(f), name_count(f), written_loops(f), executed_loops(f)])) results = [f(READINGS) for _, f in EXPRESSIONS] same = all(s == results[0] for s in results) print(f"\nall three give the same result: {'yes' if same else 'no'}") print(" " + ", ".join(f"{b} {v:.3f}" for b, v in results[0])) ``` ``` expression statement intermediate variable written loop executed loop procedural 6 6 2 2 whole-object 4 3 0 3 declarative 1 0 0 3 all three give the same result: yes east 24.674, central 22.396, north 20.210 ``` The last line is the anchor of the lesson: all three expressions give the same three regions and the same three numbers. The difference is not in the result but **in the writing**. The procedural expression holds 6 statements and binds 6 names; four of them are intermediate results and none of them enters the report. The whole-object expression comes down to 4 statements and 3 names, because rows are no longer walked: `region` and `consumption` are taken as two columns and the mean is applied to the whole column. The declarative expression is a single statement and binds no names; what is written is the recipe of the result. The real change in the second row happens not in the number but **in the unit of work**. In the procedural expression, what the code talks about is a single row named `row`, and the total and the count grow out of the fields of that row; the object of the analysis is a record. In the whole-object expression, what the code talks about is two columns named `region` and `consumption`; not even the name of a single row appears. The approach of statistical programming languages comes from here: data is handled in the unit of the **field**, not the record, and since the operation is applied to the whole field, the walking disappears entirely from the visible part of the code. ## The Loop Did Not Disappear, It Moved The last two columns of the table go in opposite directions. **Written loops** fall from 2 to 0, **executed loops** rise from 2 to 3. This is one of the most commonly misread places of abstraction: giving up writing the loop does not eliminate the loop, it only changes its place. The procedural expression writes two loops: one scans the dataset for the total and the count, the other walks the five grouped regions for the division. The whole-object expression writes no loop, but the helpers it calls run three loops — `column` scans the dataset once each time it is called and it is called twice, `group_mean` makes one scan and one merge. As the expression gets shorter, the number of passes does **not** decrease; in this example it increases. Assuming that there is a link between brevity and the number of passes is an unmeasured claim. What is gained in return is also countable: the four intermediate names in the procedural expression — `total`, `count`, `mean`, `ranked` — come down to three in the whole-object expression and to zero in the declarative expression. An intermediate name is something that has to be read and can be misused; lowering its count lowers the reading load of the expression. The price is the subject of the next section. What the measure does not count must also be written. The table counts only the statements **the person writing the analysis writes**; the bodies of the helpers do not enter the count. The four helpers and the solver together hold nine statements, and these are statements someone wrote somewhere. The gain of abstraction is not that the amount of work decreases but **that the same statements are written once and used many times**; the two columns are taken with the same `column` call in three different analyses as well. This is why the measure gives no ranking of superiority; it only shows where the load has moved. ## Five Decisions of a Single Statement The declarative expression writes four things: the source, the grouping field, the measure field and how many rows are requested. These are not all of the decisions that determine the result of the analysis. The solver below meets the same request with a complete set of defaults, and each default is changed one by one. ```python # The same declarative request, this time in a solver that carries all the defaults by name. DEFAULTS = {"missing": "count", "weight": "row", "summary": "mean", "period": "all", "direction": "descending"} def median(v): s = sorted(v) n = len(s) return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2 def solve_full(request): a = dict(DEFAULTS) a.update(request) rows = [row for row in request["source"] if (a["period"] == "all" or row["period"] == a["period"]) and (a["missing"] == "count" or row["consumption_m3"] > 0)] if a["weight"] == "subscriber": unit = {} for row in rows: unit.setdefault(row["subscriber_id"], (row[a["group"]], []))[1].append(row[a["measure"]]) rows = [{a["group"]: g, a["measure"]: sum(v) / len(v)} for g, v in unit.values()] bucket = {} for row in rows: bucket.setdefault(row[a["group"]], []).append(row[a["measure"]]) summary = {g: (sum(v) / len(v) if a["summary"] == "mean" else median(v)) for g, v in bucket.items()} ranked = sorted(summary.items(), key=lambda p: p[1], reverse=a["direction"] == "descending") return ranked[:a.get("top", len(ranked))] BASE = {"source": READINGS, "group": "region", "measure": "consumption_m3", "top": 3} CHANGES = [("unread rows", "missing", "count", "drop"), ("measure weight", "weight", "row", "subscriber"), ("summary function", "summary", "mean", "median"), ("period scope", "period", "all", "2026-01"), ("sort direction", "direction", "descending", "ascending")] baseline = dict(solve_full(BASE))["east"] y2 = align([20, 12, 12, 14, 12, 24]) print("\n" + y2(["hidden default", "default", "east m3", "changed", "east m3", "reported top three"])) values = [baseline] for name, key, before, after in CHANGES: request = {**BASE, key: after} top = solve_full(request) d = dict(solve_full({k: v for k, v in request.items() if k != "top"}))["east"] values.append(d) print(y2([name, before, f"{baseline:.3f}", after, f"{d:.3f}", " ".join(b for b, _ in top)])) print(f"\nsingle-statement expression writes {len(BASE)} decisions, leaves {len(DEFAULTS)} " f"decisions to defaults") print(f"if defaults are written the request grows {len(BASE)} -> {len(BASE) + len(DEFAULTS)} " f"keys and the expression is still 1 statement") print(f"east mean moves {max(values) - min(values):.3f} m3 between {min(values):.3f} and " f"{max(values):.3f}") ``` ``` hidden default default east m3 changed east m3 reported top three unread rows count 24.674 drop 25.593 east central north measure weight row 24.674 subscriber 24.814 east central north summary function mean 24.674 median 24.120 east central north period scope all 24.674 2026-01 25.540 east central north sort direction descending 24.674 ascending 24.674 west south north single-statement expression writes 4 decisions, leaves 5 decisions to defaults if defaults are written the request grows 4 -> 9 keys and the expression is still 1 statement east mean moves 1.473 m3 between 24.120 and 25.593 ``` A single statement writes four decisions and leaves five decisions to defaults. When the five defaults are changed one by one, the east mean becomes 25.593 — 24.814 — 24.120 — 25.540 m³ respectively instead of 24.674 m³; between the lowest and the highest there is a difference of **1.473 m³**, about six percent of the base value. The fifth does not move the number at all but changes the three reported regions completely: in descending order east, central and north are written, in ascending order west, south and north. The real observation here is this: the same five decisions **exist in the procedural expression as well**. There too the unread rows are counted, the weight is the row, and the summary is the arithmetic mean. The difference is **the number of places where the decisions can be written**. In the procedural expression the same five defaults are scattered across three statements — the loop, the division and the sorting lines — and each one sits in a place that can be changed by hand. In the single-statement expression there is no such line; a decision becomes visible only if it is added to the request as a key. When all the defaults are written explicitly, the request goes from four keys to nine keys and the expression is **still a single statement** — the cost of the record is not the statement but the key. The rule of the course finds its exact counterpart here. An abstraction shortens the work as much as it hides decisions; a function whose defaults you do not know is a function that decides on your behalf. A short expression is not bad for this reason — not knowing where a decision is written is bad. ## Summary - When the same analysis is written in three expression styles, it gives the same result: east 24.674 — central 22.396 — north 20.210 m³. - The measure of an expression is the number of statements, intermediate variables and loops: procedural 6/6/2, whole-object 4/3/0, declarative 1/0/0. - Written loops fall from 2 to 0 while executed loops rise from 2 to 3; abstraction does not remove the loop, it moves it, and in this example it adds one more. - The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed, the east mean moves within a range 1.473 m³ wide and the sort direction changes the three reported regions completely. - The same decisions exist in the procedural expression as well; what changes is the number of places where a decision can be written. ## Next Step The declarative expression came down to a single statement, but this descent has a price: a dictionary of four keys does not tell, in order, what an analysis does. The next lesson measures the form in between — the approach that writes the transformation as a chain made of individual **verbs**. The measure is from the same family: when the same seven-step transformation is written with intermediate variables, how many names are bound; when it is written as nested calls, in how many steps the reading order overlaps with the execution order; when it is written as a chain, which intermediate result's row count becomes invisible.