--- title: 'Introduction to Data Analytics' source: 'https://academia.sh/en/courses/introduction-to-data-analytics' language: en updated: '2026-10-07T04:31:12+00:00' license: 'CC BY-SA 4.0' lessons: 9 outcomes: - 'Distinguishes the kinds of analytics by the question they ask and the data they require' - 'Classifies measurement scales and shows which summary is legitimate on each scale' - 'Defines an analysis question in a measurable way and writes down how much the definition shifts the number' --- # Introduction to Data Analytics Showing that a number shifts at the moment of definition, before any calculation is made: two defensible definitions of an active subscriber giving two different numbers from the same raw set, the data each kind of analytics requires and the kind that cannot be answered with the set at hand, how many times the analysis workflow returns to the question, the same question turning into different outputs in three roles, the meaningless summary that appears when a data type is assigned wrongly, the summary a scale permits, the bias a source carries, the number of rows eliminated when the quality threshold changes, and the question that should not be asked even though the data can answer it. ## Analytics Concepts Measuring the chain from data to decision, and the decisions inside that chain, in numbers: the gap between the active subscriber counts computed from the same raw dataset with two defensible definitions, the data that descriptive, diagnostic, predictive, and prescriptive analytics require and the number of questions the dataset at hand cannot answer, how many steps the analysis workflow takes before it returns to the question, and how the same question turns into different output in the hands of a data analyst, a business analyst, and a data scientist. ### What Is Data Analytics Source: https://academia.sh/en/courses/introduction-to-data-analytics/what-is-data-analytics From a single subscriber table, the 'active subscriber count' is computed with three defensible definitions and yields three different numbers: 3,849 where the contract record is active, 3,548 where a reading was sent in the last 30 days, 4,211 where the contract is open. The totals of the first two definitions differ by only 301, yet they disagree on 645 subscribers; when the window, the one free number, is widened from 14 days to 90 days, the count rises from 1,859 to 3,750. No calculation changed, only the definition. When someone asks where a number in a report came from, the first answer is usually a description of the calculation: which rows were summed, which division was performed. This curriculum looks somewhere else. A number is often at its value not because of the calculation but because of a **definition decision** made before the calculation began. The calculation may have been done correctly and the number could still have been a different number. Data analytics is the work of making a question answerable with data and tying the resulting number to a decision. What this lesson measures is the first link of that chain: how far the decisions made while a question is made measurable move the number. The unit of measure is **the difference**: how many different numbers come out of the same data, for the same question, under two defensible decisions. - **AC1.** The dataset is the subscriber records of a **regional measurement network**: a network that collects readings from water meters, turns the readings into bills, and opens work orders for the field crew. Both the network and the records are **fiction**; every number below is the output of a model, not of a measured installation. - **AC2.** The table is ready and is used in this course as the **dataset**. Schema design, normalization, and query writing were established in the Databases curriculum and are not repeated here. - **AC3.** The reference date is day **0**. Every field ending in `Day` is the number of days counted back from that date: if `lastReadingDay` is 4, the meter sent a reading four days ago. - **AC4.** The reading cycle of the meters is monthly in the model. It is not normal for a meter to send no reading for 30 days; it is normal for it to send none for 14 days. - **AC5.** The contract status takes three values: `active`, `suspended`, `closed`. This field is the state in the **record system**; it has no necessary link to the behavior of the meter. - **AC6.** The question is this: **how many active subscribers are there in the network.** As it stands, the question is not a calculation but a request for a definition. #### What Is in the Raw Record Before answering the question, look at what is at hand. The block below builds the dataset and prints a single cross table: contract status against the time of the meter's last reading. The mismatch between the two is the source of the rest of the lesson. ```js // dataset.mjs — subscriber records of the regional measurement network; a MODEL, the records are fiction. // The reference date is day 0; fields ending in "Day" are the number of days counted back from that date. export const SEED = 20260601, SUBSCRIBERS = 4800, REGION = ["north", "south", "east", "west", "central"]; const rng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = rng(SEED); const pick = (u, d) => { let t = 0; for (const [a, p] of d) { t += p; if (u < t) return a; } return d.at(-1)[0]; }; export const RECORDS = Array.from({ length: SUBSCRIBERS }, (_, i) => { const region = REGION[Math.floor(r() * 5)]; const tier = pick(r(), [[1, 0.46], [2, 0.34], [3, 0.20]]); const connectionDay = 30 + Math.floor(r() * 2900); // The closure probability is high in the lowest tariff tier and closures pile up in recent months. const closureP = [0.16, 0.09, 0.05][tier - 1]; const closureDay = r() < closureP ? Math.min(connectionDay - 1, Math.floor(r() * r() * 420)) : null; const suspended = closureDay === null && r() < 0.085; const contract = closureDay !== null ? "closed" : suspended ? "suspended" : "active"; // Days since the meter's last reading. Some closed and suspended subscribers still // send readings; some active subscribers (faulty meter, empty home) do not. const g = contract === "active" ? (r() < 0.13 ? 31 + Math.floor(r() * 300) : Math.floor(r() * 29)) : contract === "suspended" ? (r() < 0.34 ? Math.floor(r() * 29) : 31 + Math.floor(r() * 250)) : (r() < 0.11 ? Math.floor(r() * 29) : 31 + Math.floor(r() * 500)); const consumption = Math.round((2 + tier * 4 + r() * 9) * 10) / 10; const fee = Math.round((6 + tier * tier * 7) * (0.85 + r() * 0.3)); return { i, region, tier, connectionDay, closureDay, contract, lastReadingDay: g, consumption, fee, support: r() < 0.22 ? 1 + Math.floor(r() * 3) : 0 }; }); const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`seed ${SEED}: ${SUBSCRIBERS} rows, ${Object.keys(RECORDS[0]).length} fields`); console.log("fields: " + Object.keys(RECORDS[0]).join(", ") + "\n"); const y = gen([12, 13, 11, 12, 11]); console.log(y(["contract", "subscribers", "0-30 days", "31-90 days", "90+ days"])); for (const s of ["active", "suspended", "closed"]) { const l = RECORDS.filter((k) => k.contract === s); console.log(y([s, l.length, l.filter((k) => k.lastReadingDay <= 30).length, l.filter((k) => k.lastReadingDay > 30 && k.lastReadingDay <= 90).length, l.filter((k) => k.lastReadingDay > 90).length])); } const t = (f) => RECORDS.filter(f).length; console.log(y(["total", SUBSCRIBERS, t((k) => k.lastReadingDay <= 30), t((k) => k.lastReadingDay > 30 && k.lastReadingDay <= 90), t((k) => k.lastReadingDay > 90)])); } ``` ``` seed 20260601: 4800 rows, 10 fields fields: i, region, tier, connectionDay, closureDay, contract, lastReadingDay, consumption, fee, support contract subscribers 0-30 days 31-90 days 90+ days active 3849 3376 88 385 suspended 362 121 55 186 closed 589 51 59 479 total 4800 3548 202 1050 ``` The table is read by its four corners. The top left corner is the expected case: the meters of the 3,376 subscribers whose contract is active sent a reading within the last 30 days. The top right corner is the unexpected one: the meters of 385 subscribers whose contract is active have been silent for more than 90 days. The bottom left corner is unexpected too: the meters of 51 subscribers whose contract is closed sent a reading within the last 30 days. These two corners are not a data error; both have an ordinary explanation. A silent meter may be faulty, or the home may be empty. A reading coming from a closed contract is the time that passes between the closure record being processed and the meter being removed. **The record system and the physical world move at different speeds**, and the table keeps the two in separate columns. When the question is put to this table, the answer is not single, because the phrase "active subscriber" does not say which column to look at. #### Three Definitions, Three Numbers The block below applies three definitions to the same dataset separately and prints the results side by side. All three are defensible definitions: the first looks at the record system, the second at the behavior of the meter, the third at the legal status of the contract. ```js // active.mjs — "active subscriber count" from the same raw dataset: three defensible definitions side by side. import { RECORDS, SUBSCRIBERS, SEED } from "./dataset.mjs"; const DEFINITION = [ { name: "A: contract record active", f: (k) => k.contract === "active" }, { name: "B: reading in 30 days", f: (k) => k.lastReadingDay <= 30 }, { name: "C: contract open", f: (k) => k.contract !== "closed" }, ]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const S = DEFINITION.map((t) => new Set(RECORDS.filter(t.f).map((k) => k.i))); const y = gen([26, 13, 11]); console.log(`seed ${SEED}: ${SUBSCRIBERS} subscriber records, one table, one reference date\n`); console.log(y(["definition", "subscribers", "% records"])); DEFINITION.forEach((t, a) => console.log(y([t.name, S[a].size, ((100 * S[a].size) / SUBSCRIBERS).toFixed(1)]))); const z = gen([10, 9, 11, 11, 13]); console.log("\n" + z(["pair", "diff", "only first", "only last", "disagreement"])); for (const [a, b] of [[0, 1], [2, 0], [2, 1]]) { const i1 = [...S[a]].filter((i) => !S[b].has(i)).length; const i2 = [...S[b]].filter((i) => !S[a].has(i)).length; console.log(z([`${"ABC"[a]} - ${"ABC"[b]}`, S[a].size - S[b].size, i1, i2, i1 + i2])); } // The only free number in definition B is the window length; since the reading cycle is monthly, // a window shorter than 30 days asks about a frequency the meter could never send at. const w = gen([9, 13, 12]); console.log("\n" + w(["window", "subscribers", "ratio to A"])); for (const p of [14, 21, 30, 45, 60, 90]) { const n = RECORDS.filter((k) => k.lastReadingDay <= p).length; console.log(w([`${p} days`, n, (n / S[0].size).toFixed(2)])); } ``` ``` seed 20260601: 4800 subscriber records, one table, one reference date definition subscribers % records A: contract record active 3849 80.2 B: reading in 30 days 3548 73.9 C: contract open 4211 87.7 pair diff only first only last disagreement A - B 301 473 172 645 C - A 362 362 0 362 C - B 663 714 51 765 window subscribers ratio to A 14 days 1859 0.48 21 days 2733 0.71 30 days 3548 0.92 45 days 3602 0.94 60 days 3654 0.95 90 days 3750 0.97 ``` Three definitions gave three numbers: 3,849, 3,548, and 4,211. The largest gap between them is 663 subscribers, 13.8% of the records. No calculation changed: all three definitions are a single filtering operation and none is more complicated than another. The only thing that changed is which column was looked at. The second table says something more important. The totals of A and B differ by only **301**, but the **number of subscribers the two definitions disagree on is 645**. 473 subscribers are only in A, 172 subscribers are only in B. The difference in totals looks small because most of the two-way disagreement cancels itself out. **Totals being close does not mean the two definitions count the same people.** If a report is going to carry the sentence "both methods give roughly the same result," what needs to be measured is not the difference in totals but the disagreement at the record level. #### The Window Is Also a Decision Definition B contains a number: 30 days. This number did not come from the data; it was chosen. The third table writes down the cost of the choice: when the window is cut to 14 days, the active subscriber count falls to 1,859, and when it is widened to 90 days, it rises to 3,750. Same table, same question, same calculation; the result moves by more than a factor of two. Not every value of the window is defensible. AC4 says the reading cycle is monthly; a 14-day window makes a frequency the meter is not expected to send at the criterion and counts half of the active subscribers as silent. A 90-day window falls the other way: it counts a meter that has sent no reading for three months as active. Values between 30 and 60 move the result by only 106 subscribers, so in this range the definition is sound. **The defensibility of a threshold is measured by how much the result moves around that threshold.** Moving the threshold from 14 to 21 moves the count by 874 subscribers; moving it from 45 to 60 moves it by 52. The first region is fragile, the second is stable. #### The Ranking Moves Too A single total rarely stands alone in a report. Next to the total there is a breakdown, and someone acts on the top row of that breakdown: which region to visit first, where the budget goes first. If a definition decision moves the total, it moves the ranking as well. ```js // region.mjs — the definition decision moves not only the total but the ranking: five regions, two definitions. import { RECORDS, REGION } from "./dataset.mjs"; const ratio = (l, f) => l.filter(f).length / l.length; const rows = REGION.map((b) => { const l = RECORDS.filter((k) => k.region === b); return { b, n: l.length, a: ratio(l, (k) => k.contract === "active"), c: ratio(l, (k) => k.lastReadingDay <= 30) }; }); // Rank 1 = lowest ratio, that is, the region "to look at first" in the report. const rank = (field) => new Map([...rows].sort((x, y) => x[field] - y[field]) .map((s, i) => [s.b, i + 1])); const rA = rank("a"), rB = rank("c"); const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const y = gen([9, 13, 10, 8, 10, 8, 8]); console.log(y(["region", "subscribers", "A ratio", "A rank", "B ratio", "B rank", "shift"])); for (const s of rows) { console.log(y([s.b, s.n, s.a.toFixed(4), rA.get(s.b), s.c.toFixed(4), rB.get(s.b), rB.get(s.b) - rA.get(s.b)])); } const first = (field) => [...rows].sort((x, y2) => x[field] - y2[field]); const spread = (field) => Math.max(...rows.map((s) => s[field])) - Math.min(...rows.map((s) => s[field])); console.log(`\nThe first region to look at under definition A is ${first("a")[0].b}, under B ${first("c")[0].b}`); console.log(`Under B the gap separating the first two regions is ${(first("c")[1].c - first("c")[0].c).toFixed(4)}, ` + `the spread across all five regions is ${spread("c").toFixed(4)}`); ``` ``` region subscribers A ratio A rank B ratio B rank shift north 929 0.7976 3 0.7277 3 0 south 913 0.7985 4 0.7240 1 -3 east 999 0.8248 5 0.7648 5 0 west 969 0.7905 1 0.7245 2 1 central 990 0.7970 2 0.7525 4 2 The first region to look at under definition A is west, under B south Under B the gap separating the first two regions is 0.0005, the spread across all five regions is 0.0408 ``` The head of the list changed: the first region to look at is west under definition A and south under definition B. Central moved up two places, south down three. East stayed the best region under both definitions: the shift does not happen on every row, but it happens on the row where the decision is made. The reason is written on the last row. The ratios of the five regions spread over a range of only 0.0408 under definition B. The difference between the two definitions, on the other hand, is 6.3 points across the network: the difference between 80.2% and 73.9% in the second table. **When the difference between groups is smaller than the shift a definition change produces, the ranking is a property of the definition and not of the regions.** The gap separating the first two regions under B is 0.0005; a single meter in the south sending a reading (662 instead of 661) reverses the ranking. Directing the field crew by the head of this list is directing it by a rounding difference. #### The Number Without Its Decisions The number this lesson produced is 3,548: the count of subscribers who sent a reading within the last 30 days. Three decisions stand next to it, and the number cannot be reported without all three written down: - **Which column is the criterion.** The behavior of the meter was taken as the criterion, not the contract record. Reason: if the question is really "how many subscribers can be billed," there is no bill without a reading. - **The window is 30 days.** Reason: the reading cycle is monthly, and between 30 and 60 the result moves by 106 subscribers, so the choice does not determine the result in this range. - **Contract status was not filtered.** The 51 subscribers whose contract is closed stayed inside the count because they sent a reading. Reason: in the question of whether a bill can be issued these are counted too; that the closure was not processed is a separate problem. When these three lines are not written, what is left is the number 3,548, and that number is not a finding but a choice. **A number without its decisions counts as unmeasured.** This rule will continue throughout the curriculum and will be applied in the same form in every lesson: the reported number, the decisions taken while producing it, the difference that appears when a decision is changed. A second conclusion follows, and it is more uncomfortable. There is no such thing as "the data speaks." In this table there are three correct numbers and none of them refutes another; what determines which one will be reported is not the data but what the question is being asked for. If a meter replacement list is to be drawn from the same table, the 645 subscribers on which A and B disagree are the target; if a bill run is to be made, it is B; if a contract count is to be reported, it is C. The job of analytics is not to choose one of these three numbers but to write down which of them corresponds to which question. #### Summary - Data analytics is the work of making a question measurable; a number is often determined not during the calculation but during the definition. - From a single subscriber table, "active subscriber" gives three different numbers under three defensible definitions: 3,849 (contract record active), 3,548 (reading in the last 30 days), 4,211 (contract open). - Totals being close does not show agreement: the totals of A and B differ by 301, but the number of subscribers they disagree on at the record level is 645. - A threshold inside a definition is also a decision; when the window is widened from 14 days to 90 days, the count rises from 1,859 to 3,750. The defensibility of a threshold is measured by how much the result fluctuates around it. - A definition decision moves the ranking as well: the first region to look at is west under definition A and south under definition B, and under B a single meter's reading separates the first two regions. - If a number does not have next to it the decision that produced it, that number is a choice and not a finding; a number without its decisions counts as unmeasured. #### Next Step This lesson made a single question measurable: **how many active subscribers are there.** The question itself was not questioned. Yet the questions that can be put to the same table are not of a single kind. "How many subscribers are there" describes the past; "why did the subscriber count fall" asks for a cause; "how many subscribers will there be next quarter" asks for a forecast; "which subscriber should be visited first" recommends an action. All four can be put to the same table, but all four cannot be answered with the same data. The next lesson separates these four kinds of question and counts how many in a set of questions can be answered with the dataset at hand. ### Types of Analytics Source: https://academia.sh/en/courses/introduction-to-data-analytics/types-of-analytics The 16 questions put to the same subscriber table are split into descriptive, diagnostic, predictive, and prescriptive, and the number of answerable questions moves between 2 and 7 depending on two decisions. Counting derived fields as present opens 5 questions, and raising the history depth required for a predictive question from 3 periods to 12 closes 2 questions. The four questions of the prescriptive type open under none of the four decision combinations, because the table holds a record of what happened and no record of what was tried. The previous lesson made a single question measurable: how many active subscribers there are in the network. The question itself was not questioned. Yet the questions that can be put to the same table are not of one kind, and when the kind changes, what is required of the table changes too. "How many subscribers are there" summarizes what stands in the record; "why did the subscriber count fall" asks for a cause; "how many subscribers will there be next month" asks for a forecast; "which subscriber should be visited first" asks for the result of an action. This lesson separates the four kinds and measures **which data each kind requires**. The number measured is the count of questions in a question set that can be answered with the data at hand. That number also has a definition, and the definition depends on two decisions; when the two decisions change, the number of answers over the same 16 questions moves between 2 and 7. - **AC7.** The data at hand is the subscriber table of the previous lesson; next to it, a **four-month summary** of the meter readings has been added. The history depth is four periods. Both the dataset and the questions are **fiction**. - **AC8.** The question set consists of 16 questions, four from each type. - **AC9.** A question counts as **answerable** when every field it requires either stands in the dataset or is derivable, and the history depth it requires is on hand. Whether the answer is **correct** is a separate question; what is measured here is the sufficiency of the data, not the quality of the answer. - **AC10.** A **derivable field** does not stand on disk but is computed from fields that do. The computation requires a decision: the previous lesson's question "which subscribers go into the denominator" comes up again in every derivation. - **AC11.** Two decisions are measured. The first: whether a derivable field counts as "present" (strict reading) or does not (loose reading). The second: how many periods of history are required at minimum for a predictive question to count as answerable, 3 or 12. #### Four Kinds of Questions The four types correspond to four different tenses. **Descriptive analytics** asks *what happened*. It summarizes what stands in the record: it counts, sums, takes ratios, breaks down. All of the work in the previous lesson was descriptive. The data it requires is the thing standing in the record itself; this makes its data request the lightest of the four types. **Diagnostic analytics** asks *why it happened*. It asks for a cause, and if the cause does not stand in the record as a field, it asks for a comparison that can tell the cause apart. If the reason for closure was not recorded, comparing the subscribers who closed with those who did not yields an **association**; saying that the association is the cause is a separate decision and does not come out of the data. **Predictive analytics** asks *what will happen*. It rests on the assumption that a pattern observed in the past will continue, and therefore it requires enough history to see the pattern. The question "how many periods are enough" has no answer that comes from the data; it is a threshold decision, and it is the second measure of this lesson. **Prescriptive analytics** asks *what should be done*. It requires the result of an action: what happens if a discount is given, what changes if a visit is made. If the action was never tried, its result is not in the record; if it was tried but the result was not recorded, it is not there either. #### The Data at Hand and the Question Set ```js // inventory.mjs — the data at hand and a question set; each question is a type and a data request. // A MODEL: both the fields and the questions come from the fiction of the regional measurement network. export const DEPTH = 4; // how many periods (monthly) of history are on hand export const HAVE = new Set(["subscriber id", "region", "tariff tier", "connection day", "closure day", "contract status", "last reading day", "monthly consumption", "monthly fee", "support request count"]); // A derivable field does not stand on disk but is computed from fields that do; the computation requires a decision. export const DERIVABLE = new Map([ ["subscriber age", "connection day"], ["closure rate", "closure day + contract status"], ["consumption per subscriber", "monthly consumption + subscriber id"], ["consumption change rate", "monthly consumption (four periods)"], ]); export const QUESTIONS = [ { s: "how many subscribers sent readings in the last 30 days", t: "descriptive", f: ["last reading day"], d: 1 }, { s: "which region has high consumption per subscriber", t: "descriptive", f: ["region", "consumption per subscriber"], d: 1 }, { s: "subscriber distribution by tariff tier", t: "descriptive", f: ["tariff tier"], d: 1 }, { s: "how did total consumption change over four months", t: "descriptive", f: ["consumption change rate"], d: 4 }, { s: "why did subscribers close their contracts", t: "diagnostic", f: ["closure reason"], d: 1 }, { s: "did the drop in readings come from an outage", t: "diagnostic", f: ["outage record"], d: 1 }, { s: "is the rise in support due to a tariff change", t: "diagnostic", f: ["support request count", "tariff change history"], d: 1 }, { s: "does the closure rate vary by tariff tier", t: "diagnostic", f: ["closure rate", "tariff tier"], d: 1 }, { s: "how many subscribers will close next month", t: "predictive", f: ["closure rate"], d: 1 }, { s: "what will total consumption be next quarter", t: "predictive", f: ["consumption change rate"], d: 4 }, { s: "which subscriber's meter will go silent next month", t: "predictive", f: ["reading history"], d: 1 }, { s: "what will the support request count be next month", t: "predictive", f: ["support history"], d: 1 }, { s: "which subscriber should get a field visit first", t: "prescriptive", f: ["intervention and outcome"], d: 1 }, { s: "will changing the tier reduce closures", t: "prescriptive", f: ["intervention and outcome", "tariff change history"], d: 1 }, { s: "which region should get the meter renewal budget", t: "prescriptive", f: ["intervention and outcome"], d: 1 }, { s: "should a closing subscriber be offered a discount", t: "prescriptive", f: ["intervention and outcome"], d: 1 }, ]; export const TYPES = ["descriptive", "diagnostic", "predictive", "prescriptive"]; // Two decisions: whether a derived field counts as "present", and how many periods of history a predictive question requires. export function missing(question, loose) { return question.f.filter((x) => !HAVE.has(x) && !(loose && DERIVABLE.has(x))); } export function answerable(question, loose, forecastDepth) { const required = question.t === "predictive" ? Math.max(question.d, forecastDepth) : question.d; return missing(question, loose).length === 0 && required <= DEPTH; } const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`${HAVE.size} fields and ${DEPTH} periods of history on hand; ` + `${DERIVABLE.size} fields are derivable\n`); const y = gen([55, 14, 9]); console.log(y(["question", "type", "periods"]) + " missing field (strict reading)"); for (const q of QUESTIONS) { const e = missing(q, false); console.log(y([q.s, q.t, q.d]) + " " + (e.length ? e.join(", ") : "-")); } } ``` ``` 10 fields and 4 periods of history on hand; 4 fields are derivable question type periods missing field (strict reading) how many subscribers sent readings in the last 30 days descriptive 1 - which region has high consumption per subscriber descriptive 1 consumption per subscriber subscriber distribution by tariff tier descriptive 1 - how did total consumption change over four months descriptive 4 consumption change rate why did subscribers close their contracts diagnostic 1 closure reason did the drop in readings come from an outage diagnostic 1 outage record is the rise in support due to a tariff change diagnostic 1 tariff change history does the closure rate vary by tariff tier diagnostic 1 closure rate how many subscribers will close next month predictive 1 closure rate what will total consumption be next quarter predictive 4 consumption change rate which subscriber's meter will go silent next month predictive 1 reading history what will the support request count be next month predictive 1 support history which subscriber should get a field visit first prescriptive 1 intervention and outcome will changing the tier reduce closures prescriptive 1 intervention and outcome, tariff change history which region should get the meter renewal budget prescriptive 1 intervention and outcome should a closing subscriber be offered a discount prescriptive 1 intervention and outcome ``` The last column splits the missing items into two groups. One group is **derivable** fields: consumption per subscriber, closure rate, consumption change rate. They do not stand on disk but are computed from fields that do. The other group is the fields that are **not there at all**: closure reason, outage record, reading history, support history, tariff change history, intervention and outcome. These cannot be produced by calculation, because they were not recorded at the source. The distinction matters, because the first group is open to a decision and the second is not. Whether a derivable field counts as "present" is a reading preference. There is no such preference as counting a field that does not exist at all as present; until that data is collected, the question is unanswered. #### Two Decisions, Two Numbers ```js // decision.mjs — the same 16 questions, the same table, two decisions: the answerable counts side by side. import { QUESTIONS, TYPES, DEPTH, answerable, missing } from "./inventory.mjs"; const COMBOS = [ { name: "strict / 3 periods", g: false, o: 3 }, { name: "loose / 3 periods", g: true, o: 3 }, { name: "strict / 12 periods", g: false, o: 12 }, { name: "loose / 12 periods", g: true, o: 12 }, ]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const y = gen([20, 13, 12, 12, 14, 8]); console.log(`${DEPTH} periods of history on hand; the numbers below are counts of answerable questions\n`); console.log(y(["decision combo", ...TYPES, "total"])); const totals = COMBOS.map((b) => { const c = TYPES.map((t) => QUESTIONS.filter((q) => q.t === t && answerable(q, b.g, b.o)).length); console.log(y([b.name, ...c, c.reduce((x, v) => x + v, 0)])); return c.reduce((x, v) => x + v, 0); }); console.log(`\nsame 16 questions: lowest ${Math.min(...totals)}, highest ${Math.max(...totals)} ` + `answers; difference ${Math.max(...totals) - Math.min(...totals)} questions`); const opened = QUESTIONS.filter((q) => !answerable(q, false, 3) && answerable(q, true, 3)); console.log(`when derived fields count as present, ${opened.length} questions open: ` + opened.map((q) => q.t).join(", ")); for (const t of TYPES) { const none = QUESTIONS.filter((q) => q.t === t) .every((q) => COMBOS.every((b) => !answerable(q, b.g, b.o))); if (none) { const n = QUESTIONS.filter((q) => q.t === t); const shared = [...new Set(n.flatMap((q) => missing(q, true)))]; console.log(`\n${t} type opens under none of the four decisions: ${n.length} questions, 0 answers`); console.log(`missing fields that no decision closes: ${shared.join(", ")}`); } } ``` ``` 4 periods of history on hand; the numbers below are counts of answerable questions decision combo descriptive diagnostic predictive prescriptive total strict / 3 periods 2 0 0 0 2 loose / 3 periods 4 1 2 0 7 strict / 12 periods 2 0 0 0 2 loose / 12 periods 4 1 0 0 5 same 16 questions: lowest 2, highest 7 answers; difference 5 questions when derived fields count as present, 5 questions open: descriptive, descriptive, diagnostic, predictive, predictive prescriptive type opens under none of the four decisions: 4 questions, 0 answers missing fields that no decision closes: intervention and outcome, tariff change history ``` Same table, same 16 questions; the number of answerable questions is between 2 and 7. The difference is 5 questions, nearly a third of the set. Neither of the two decisions touched the data; both changed the answer to the question "what do we have." The first decision alone has the largest effect. Counting derived fields as present opens 5 questions: two descriptive, one diagnostic, two predictive. This decision is not free. If the closure rate is to be derived, a denominator must be chosen, and the previous lesson showed that the denominator has three defensible values. So saying "the derivable field exists" means adding a new definition decision to the answer to the question. **The loose reading opens the question but increases the number of decisions that must be written down along with it.** The second decision moves only the predictive row: when the depth requirement goes from 3 to 12, two predictive questions close and the total falls from 7 to 5. Four periods of history are on hand; this passes the requirement of 3 and fails the requirement of 12. The requirement itself is not read from the data. It is possible to argue that four periods will not show a yearly fluctuation, and it is also possible to argue that four periods are enough for a monthly forecast. What needs to be written in the report is which one was chosen. The diagnostic row rises to 1 at most under the four decisions. The question that opens is whether the closure rate varies by tariff tier, and what it yields is an association: at which tier closure is more frequent. The conclusion that the tier is the cause of closure does not come out of this table. What could tell the cause apart is either a record of the closure reason or a comparison that changes the cause alone; the dataset has neither. **That a diagnostic question counts as answerable does not mean the answer will be a cause.** #### The Gap That Does Not Close The prescriptive column is zero in all four rows. This is the most robust number of the lesson, because it is not sensitive to any decision. The shared missing fields are two, and one of them is decisive: **intervention and outcome.** The table is a record of what happened. Which subscriber's contract closed, when it closed, how much the subscriber consumed before closing: all of it is written. What is not written is this: whether anything was done to that subscriber, and if so what was done, and what happened after it was done. Whether a discount offer reduces closures can be read only from a setup in which some subscribers were given the offer and some were not, and the outcomes of the two groups were recorded separately. If no such setup was built, this question stays unanswered however large the dataset grows. The rule that follows is this: **collecting data is an analysis decision, and it is made before the analysis.** A network that answers prescriptive questions is a network that keeps intervention records knowing it will ask those questions. The type of record on hand determines which questions can be asked; when the question set grows, the record does not grow by itself. #### Summary - The four types of analytics ask four different questions: what happened (descriptive), why it happened (diagnostic), what will happen (predictive), what should be done (prescriptive). A change in type changes the data required from the table. - Over the same 16 questions and the same table, the number of answerable questions moves between 2 and 7; two decisions produce this difference without touching the data at all. - Counting a derived field as present opens 5 questions but every derivation adds a new definition decision; when the predictive depth requirement goes from 3 to 12, two questions close. - A diagnostic question that counts as answerable yields an association; the record or comparison that could tell the cause apart is not in the dataset. - The four questions of the prescriptive type open under none of the four decision combinations, because the dataset has no record of intervention and outcome; the decision to collect data is made before the analysis. #### Next Step This lesson and the previous one together may leave an impression: first a question is chosen, then its type is determined, then one checks whether the data is enough, then the calculation is done. It looks like a sequential list. In reality this list does not run once from start to finish. The window decision in the previous lesson became defensible only after the data was looked at; the depth requirement in this lesson gained a number only after the question set was written. In both, something learned in a later step changed a decision written in an earlier step. The next lesson builds the analysis workflow in five steps and counts how many times the workflow turns back. ### Analysis Workflow Source: https://academia.sh/en/courses/introduction-to-data-analytics/analysis-workflow A workflow of five steps (question, data, analysis, finding, communication) is run together with 12 decisions: 10 of the decisions are seen after the step they belong to. When decisions are written as they are seen, the workflow produces 10 rework loops and 33 step visits; when every decision is written in its own step, it drops to 5 loops and 15 visits, and when a preview is added, to 1 loop and 8 visits. The one remaining loop does not close, because its content is not in the raw data but in the output of the analysis. When the accounting decision changes, the same workflow gives 15 visits instead of 33. The previous two lessons may have left an impression: first a question is chosen, then its type is determined, then one checks whether the data is enough, then the calculation is done. It looks like a sequential list. In reality this list does not run once from start to finish. The 30-day window in the first lesson became defensible only after the table was looked at and the reading cycle was seen to be monthly; the depth requirement in the second lesson gained a number only after the question set was written. This lesson builds the workflow in five steps and shows with a number that the workflow is not linear. The measure is **the number of rework loops**: how many times it was necessary to return to an earlier step because of a decision seen in a later step. A second number is placed next to it: how many times in total the five steps of the workflow were run. - **AC12.** The workflow is five steps: **question**, **data**, **analysis**, **finding**, **communication**. Both the workflow and the decisions are **fiction**; the decisions are taken from the subscriber counts of the previous two lessons. - **AC13.** A **rework loop** is returning to the step a decision belongs to, when that decision is seen in a later step, and rerunning the steps in between. - **AC14.** A **step visit** is counted as follows: if the workflow runs once from start to finish it is 5 visits; a loop from step $f$ back to step $a$ adds $f - a + 1$ visits. This is an **accounting decision** and will be changed at the end of the lesson. - **AC15.** A decision counts as **writable in advance** if it can be written in the step it belongs to, without looking at the output of a later step. If it cannot be written, the content of the decision is learned only in a later step. - **AC16.** Time is not measured. What is counted is loops and visits. #### Five Steps The **question** step makes the thing to be asked measurable: which definition, which reference date, which decision the result will feed, who will read it. The whole subject of the first lesson was this step. The **data** step finds the fields the question asks for and sees the real state of the dataset: which field is filled, which row is duplicated, which value is missing. The inventory of the second lesson is the output of this step. The **analysis** step filters, groups, and summarizes; the number comes out here. Table design and query writing were established in the Databases curriculum and are not repeated here; what is measured in this step is not syntax but the filtering and grouping decisions. The **finding** step turns the number into a sentence: breakdown, ranking, comparison. The **communication** step ties the finding to a decision and writes it for the party who will read it. The last two steps do not produce decisions, they consume them, and for exactly this reason a missing decision seen there sends the workflow back to the very beginning. #### Where Decisions Are Born, Where They Are Seen ```js // workflow.mjs — a five-step analysis workflow and the decisions that send it back. A MODEL: // the decisions are taken from the subscriber counts of the previous two lessons, the workflow is fiction. export const STEPS = ["question", "data", "analysis", "finding", "communication"]; // home: the step in which the decision has to be written. seen: the step in which the decision is noticed. // advance: whether the decision can be written in its own step without looking at the data. export const DECISIONS = [ { name: "active subscriber definition", home: 1, seen: 1, advance: true }, { name: "reference date", home: 1, seen: 1, advance: true }, { name: "reading window 30 days", home: 1, seen: 2, advance: false }, { name: "readings from closed contracts", home: 1, seen: 2, advance: false }, { name: "region breakdown requested", home: 1, seen: 4, advance: true }, { name: "who will read the report", home: 1, seen: 5, advance: true }, { name: "comparison period", home: 1, seen: 5, advance: true }, { name: "rows with empty region", home: 2, seen: 3, advance: false }, { name: "duplicate subscriber record", home: 2, seen: 3, advance: false }, { name: "tier field: text or number", home: 2, seen: 3, advance: true }, { name: "ranking criterion: ratio or count", home: 3, seen: 4, advance: true }, { name: "ranking is volatile near threshold", home: 3, seen: 4, advance: false }, ]; // Decisions whose content is learned by looking at the raw data: a preview pulls these forward. export const FROM_DATA = new Set(["reading window 30 days", "readings from closed contracts", "rows with empty region", "duplicate subscriber record"]); // A rework loop is going back from the seen step to the home step and rerunning what is in between. export const distance = (k) => k.seen - k.home; export const cost = (k) => k.seen - k.home + 1; export const policy = (written, preview = false) => { const d = DECISIONS.filter((k) => distance(k) > 0 && !(written && k.advance) && !(preview && FROM_DATA.has(k.name))); return { loops: d.length, list: d, visits: STEPS.length + (preview ? 1 : 0) + d.reduce((s, k) => s + cost(k), 0) }; }; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`workflow of ${STEPS.length} steps: ${STEPS.join(" -> ")}`); console.log(`${DECISIONS.length} decisions need to be written\n`); const y = gen([40, 12, 17, 9, 12]); console.log(y(["decision", "belongs to", "seen at", "distance", "in advance"])); for (const k of DECISIONS) { console.log(y([k.name, `${k.home} ${STEPS[k.home - 1]}`, `${k.seen} ${STEPS[k.seen - 1]}`, distance(k), k.advance ? "yes" : "no"])); } const g = DECISIONS.filter((k) => distance(k) > 0); console.log(`\n${g.length} of ${DECISIONS.length} decisions are seen after the step they belong to`); console.log(`of these, ${g.filter((k) => FROM_DATA.has(k.name)).length} are learned only by ` + `looking at the raw data: ${g.filter((k) => FROM_DATA.has(k.name)).map((k) => k.name).join("; ")}`); } ``` ``` workflow of 5 steps: question -> data -> analysis -> finding -> communication 12 decisions need to be written decision belongs to seen at distance in advance active subscriber definition 1 question 1 question 0 yes reference date 1 question 1 question 0 yes reading window 30 days 1 question 2 data 1 no readings from closed contracts 1 question 2 data 1 no region breakdown requested 1 question 4 finding 3 yes who will read the report 1 question 5 communication 4 yes comparison period 1 question 5 communication 4 yes rows with empty region 2 data 3 analysis 1 no duplicate subscriber record 2 data 3 analysis 1 no tier field: text or number 2 data 3 analysis 1 yes ranking criterion: ratio or count 3 analysis 4 finding 1 yes ranking is volatile near threshold 3 analysis 4 finding 1 no 10 of 12 decisions are seen after the step they belong to of these, 4 are learned only by looking at the raw data: reading window 30 days; readings from closed contracts; rows with empty region; duplicate subscriber record ``` The table is read with two columns. The left column says where the decision belongs, the right column says where it is seen. Of the twelve decisions, only two are seen where they are born: the active subscriber definition and the reference date. The remaining ten emerge after the step they belong to. The distance column says that the loops are not equal. Readings arriving from closed contracts send the workflow back one step; noticing in the communication step who will read the report sends it back four steps. Where the second one is seen has nothing to do with what the question is; it has to do with the step of the workflow at which someone first asks "who is this for." The five decisions seen in the last two steps are also worth noting. The finding and communication steps produce no decisions at all: 4 and 5 never appear in the `belongs to` column of the table. But five decisions are **seen** there. The steps that consume decisions are the steps that notice the absence of the decisions they did not produce. #### Three Working Modes The same workflow, the same 12 decisions, one difference: when the decisions are written. In the first mode every decision is written the moment it is seen. In the second mode, in each step, the decisions that belong to that step and can be written in advance are written before leaving the step. The third mode adds one thing to the second: before leaving the question step, the raw data is looked at once. This look costs one step visit and is not an analysis; it only shows whether the fields are filled and which values occur. ```js // loop.mjs — the same workflow, two working modes: rework loops and step visits side by side. import { STEPS, DECISIONS, FROM_DATA, distance, cost, policy } from "./workflow.mjs"; const A = policy(false), B = policy(true), C = policy(true, true); const gen = (C) => (h) => h.map((v, j) => (j ? String(v).padStart(C[j]) : String(v).padEnd(C[0]))).join(""); const y = gen([34, 14, 11, 13, 11]); console.log(y(["working mode", "rework loops", "visits", "per step", "farthest"])); for (const [name, p] of [["decisions written when seen", A], ["decisions written in their step", B], ["written in step + preview", C]]) { console.log(y([name, p.loops, p.visits, (p.visits / STEPS.length).toFixed(1), Math.max(...p.list.map(distance))])); } const z = gen([17, 10, 10, 10]); console.log("\n" + z(["returned to", "mode 1", "mode 2", "mode 3"])); for (let a = 1; a <= STEPS.length; a++) { const s = (p) => p.list.filter((k) => k.home === a).length; console.log(z([`${a} ${STEPS[a - 1]}`, s(A), s(B), s(C)])); } console.log(`\ntotal visits ${A.visits} -> ${B.visits}, rework loops ${A.loops} -> ${B.loops}`); const long = A.list.filter((k) => distance(k) >= 3); console.log(`in mode 1, ${long.length} loops with a distance of 3 or more take up ` + `${long.reduce((s, k) => s + cost(k), 0)} of the ${A.visits - STEPS.length} extra visits`); console.log(`the ${B.loops} loops left in mode 2: ${B.list.map((k) => k.name).join("; ")}`); console.log(`these ${B.loops} decisions are all of the ${DECISIONS.filter((k) => !k.advance).length} ` + `decisions that cannot be written in advance; they are learned by looking, not by writing`); console.log(`\nthe preview pulls ${FROM_DATA.size} decisions forward, at a cost of 1 visit: ` + `${B.loops} loops fall to ${C.loops}, ${B.visits} visits fall to ${C.visits}`); console.log(`the only loop left in mode 3: ${C.list.map((k) => k.name).join("; ")} ` + `— its content is in the output of the analysis, not in the raw data`); // A second accounting decision: a rework loop reruns only the step it belongs to, not the steps in between. const cheap = (p) => STEPS.length + p.loops; console.log(`\nif the accounting decision changes (only the home step reruns): ` + `mode 1 ${cheap(A)} visits, mode 2 ${cheap(B)} visits`); console.log(`mode 1 with cheap accounting ${cheap(A)}, mode 2 with full accounting ${B.visits}: ` + `the same number, a different workflow`); ``` ``` working mode rework loops visits per step farthest decisions written when seen 10 33 6.6 4 decisions written in their step 5 15 3.0 1 written in step + preview 1 8 1.6 1 returned to mode 1 mode 2 mode 3 1 question 5 2 0 2 data 3 2 0 3 analysis 2 1 1 4 finding 0 0 0 5 communication 0 0 0 total visits 33 -> 15, rework loops 10 -> 5 in mode 1, 3 loops with a distance of 3 or more take up 14 of the 28 extra visits the 5 loops left in mode 2: reading window 30 days; readings from closed contracts; rows with empty region; duplicate subscriber record; ranking is volatile near threshold these 5 decisions are all of the 5 decisions that cannot be written in advance; they are learned by looking, not by writing the preview pulls 4 decisions forward, at a cost of 1 visit: 5 loops fall to 1, 15 visits fall to 8 the only loop left in mode 3: ranking is volatile near threshold — its content is in the output of the analysis, not in the raw data if the accounting decision changes (only the home step reruns): mode 1 15 visits, mode 2 10 visits mode 1 with cheap accounting 15, mode 2 with full accounting 15: the same number, a different workflow ``` The five-step workflow runs 33 times in the first mode. 6.6 visits per step means that describing the workflow as "a five-step process" describes the diagram and not the work. In the second mode the number falls to 15, to 3.0 per step. The difference of 18 visits between them is gained without collecting any additional data, only by changing when the decisions are written. Where the gain is collected is written in the third group of lines. The three loops with a distance of three or more take up 14 of the 28 extra visits in the first mode: a third of the loops carry half of the cost. All three are the same kind of decision: whether a region breakdown will be requested, who will read the report, which period it will be compared with. None of them asks anything about the data; all three ask whom and what the question is being asked for, and all three can be written in the first step without looking at any table. #### The Loop That Does Not Go to Zero In the second mode the rework loops stop at five, and all five are a property of the dataset: the reading window, readings from closed contracts, rows with an empty region, duplicate subscriber records, the ranking being volatile near the threshold. None of them can be written at a desk; that the reading cycle is monthly was seen when the cross table of the first lesson was printed, and that the ranking turns on a difference of 0.0005 was seen when the region table was printed. The third mode pulls four of these five forward. The preview pays one visit and in return brings 15 visits down to 8 and five loops down to **one**. The first two steps are never returned to. The gain comes from all four decisions being visible in the raw data: an empty field, a duplicated identifier, or a date distribution is seen by looking, without any calculation. The one remaining loop does not close, and cannot. The ranking being volatile near the threshold does not stand in the raw data; it appears after the analysis has run and the region ratios have been computed. **At which step a decision can be learned depends on the content of that decision and cannot be changed by the working arrangement.** The workflow is not linear; it is not linear not because of disorder but because of **the order of information**. This is why the advice "define the question correctly first" is only half a piece of advice: part of the definition can be written only after the data is seen, and part only after the calculation has run. What can be done is to pull each decision forward to the earliest step at which it can be learned, and to leave room, knowing that the rest will loop back. The last two lines apply the same test to the lesson's own number. AC14 was an accounting decision: a loop reruns all the steps in between. A second accounting decision is also defensible: only the step the decision belongs to reruns, and the steps in between are left alone if they have not been broken. Under this decision the first mode produces 15 visits and the second mode 10. That is, **the first mode comes out at 15 with cheap accounting, and the second mode comes out at 15 with full accounting.** Two different working modes, with two different accounting decisions, give the same number. Unless it is written which 15 it is, this number says nothing; the lesson's own measure is also subject to the rule of the course. #### Summary - The analysis workflow is five steps: question, data, analysis, finding, communication. The finding and communication steps consume decisions rather than produce them; but they are also the steps that notice a missing decision. - In this run, 10 of the 12 decisions are seen after the step they belong to; only two are seen where they are born. - When decisions are written as they are seen, the workflow produces 10 rework loops and 33 step visits, 6.6 per step. When every decision is written in its own step it falls to 5 loops and 15 visits, and when a preview of the raw data is added, to 1 loop and 8 visits. - The gain is collected in the long loops: the three loops with a distance of three or more take up 14 of the extra visits, and all three concern whom the question is asked to, not the data. - The content of the one loop that does not close is in the output of the analysis, not in the raw data; at which step a decision can be learned cannot be changed by the working arrangement. - The number of visits also depends on an accounting decision: the first mode gives 15 with cheap accounting and the second mode gives 15 with full accounting; the number cannot be read without writing which 15 it is. #### Next Step This lesson assumed that a single person ran the workflow. In reality the steps of the workflow are often divided among people who have different jobs, and where the division falls changes what the result will be. The same question, whether the active subscriber count is falling, turns into a number in the hands of someone charged with measuring, into a threshold in the hands of someone charged with tying the decision, and into a prediction interval in the hands of someone charged with generalizing to the future. All three read the same table, and all three deliver a different output. The next lesson separates these three job descriptions and prints side by side how the same question turns into three different numbers in three hands. ### Analyst Roles Source: https://academia.sh/en/courses/introduction-to-data-analytics/analyst-roles A one-sentence question is given to three job descriptions, and the same table produces three different outputs: the measuring role reports -228 subscribers in 90 days (-5.14%), the decision-binding role gets -3.34% with the fee criterion and, since this falls short of the -5% threshold, takes no action, and the generalizing role gives two forecasts for one year ahead, between 2,191 and 3,286. Fee per subscriber has risen 1.89% from the same rows; of the three correct numbers, two show a drop and one shows a rise. Behind a one-sentence question stand 10 written decisions. The previous lesson assumed that a single person ran the workflow. In reality the steps of the workflow are often divided among people who have different jobs, and where the division falls changes what the result will be. The same question turns into a number in one pair of hands, into a threshold in another, and into a prediction interval in a third. This lesson separates three job descriptions and gives all three the same question: **is the subscriber count falling across the network, and what should be done.** What is measured is how different the three outputs from the same table are. Role names are treated here as **job descriptions**, not as titles: what defines a role is not its name but the acceptance criterion that says when its work is done. - **AC17.** The records are the records of the first lesson: the same seed, the same 4,800 rows, the same numbers. The dataset is **fiction**. - **AC18.** Role names are job descriptions. The same name may describe a different job somewhere else; what is measured is not the name but the **acceptance criterion**: when the output is such and such, the job counts as done. - **AC19.** The question is given to all three in a single sentence, and all three read the same table. Nobody is given additional data. - **AC20.** The fee field is a **fee unit**; the scale is the scale of the model, not any currency. - **AC21.** The comparison period is 90 days and the action threshold is -5%. Both are decisions, and both were written **before** any number was calculated. #### Three Job Descriptions The **measuring job description** (data analyst) is done when it delivers the number that was asked for, together with its definition and decisions. It does not change the question. Whether the thing asked is the right question is outside this description; the product of this role is a number and how that number was produced. The **decision-binding job description** (business analyst) is done when the number is tied to an action. For that, the number must have a threshold and something to be done when the threshold is crossed. This description includes the **authority to change the criterion of the question**: if the quantity asked about cannot be tied to an action, it is replaced with a quantity that can. The **generalizing job description** (data scientist) is done when the finding has been carried from the observed period to a period not yet seen, or to rows not yet seen. Carrying requires an assumption, and the assumption is delivered as part of the output; a generalization without an assumption is unfinished work in this description. The three descriptions do not do the same job, and none of them is a more advanced version of another. The difference between them is not a difference of skill but of **where the work ends**. #### The Single Table the Three Read ```js // series.mjs — the subscriber records of the first lesson, same seed and same numbers; a MODEL. // The reference date is day 0; fields ending in "Day" are the number of days counted back from that date. export const SEED = 20260601, SUBSCRIBERS = 4800, REGION = ["north", "south", "east", "west", "central"]; const rng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = rng(SEED); const pick = (u, d) => { let t = 0; for (const [a, p] of d) { t += p; if (u < t) return a; } return d.at(-1)[0]; }; export const RECORDS = Array.from({ length: SUBSCRIBERS }, (_, i) => { const region = REGION[Math.floor(r() * 5)]; const tier = pick(r(), [[1, 0.46], [2, 0.34], [3, 0.20]]); const connectionDay = 30 + Math.floor(r() * 2900); // The closure probability is high in the lowest tariff tier and closures pile up in recent months. const closureP = [0.16, 0.09, 0.05][tier - 1]; const closureDay = r() < closureP ? Math.min(connectionDay - 1, Math.floor(r() * r() * 420)) : null; const suspended = closureDay === null && r() < 0.085; const contract = closureDay !== null ? "closed" : suspended ? "suspended" : "active"; // Days since the meter's last reading. Some closed and suspended subscribers still // send readings; some active subscribers (faulty meter, empty home) do not. const g = contract === "active" ? (r() < 0.13 ? 31 + Math.floor(r() * 300) : Math.floor(r() * 29)) : contract === "suspended" ? (r() < 0.34 ? Math.floor(r() * 29) : 31 + Math.floor(r() * 250)) : (r() < 0.11 ? Math.floor(r() * 29) : 31 + Math.floor(r() * 500)); const consumption = Math.round((2 + tier * 4 + r() * 9) * 10) / 10; const fee = Math.round((6 + tier * tier * 7) * (0.85 + r() * 0.3)); return { i, region, tier, connectionDay, closureDay, contract, lastReadingDay: g, consumption, fee, support: r() < 0.22 ? 1 + Math.floor(r() * 3) : 0 }; }); // A subscriber whose contract was open d days ago: connected by that date and not yet closed. export const openAt = (d) => RECORDS.filter((k) => k.connectionDay >= d && (k.closureDay === null || k.closureDay < d)); export const point = (d) => { const l = openAt(d), u = l.reduce((s, k) => s + k.fee, 0); return { d, n: l.length, u, per: u / l.length }; }; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`seed ${SEED}: ${SUBSCRIBERS} subscriber records; the series below is read from a single table\n`); const y = gen([11, 13, 15, 20]); console.log(y(["days ago", "subscribers", "total fee", "fee per subscriber"])); for (const d of [90, 60, 30, 0]) { const p = point(d); console.log(y([d, p.n, p.u, p.per.toFixed(2)])); } } ``` ``` seed 20260601: 4800 subscriber records; the series below is read from a single table days ago subscribers total fee fee per subscriber 90 4439 139508 31.43 60 4426 139657 31.55 30 4377 138739 31.70 0 4211 134843 32.02 ``` Four rows, three columns. The number of subscribers with an open contract goes from 4,439 to 4,211 in 90 days, and the drop is accelerating: 13 in the first 30 days, 166 in the last 30. Total fee falls in the same direction between the ends but more slowly, and in the first 30 days it rises on its own. The third column goes the opposite way: fee per subscriber rises at every step. All three columns came from the same rows, and all three are correct. Which of them is the answer to "is the subscriber count falling" is not written in the table. #### Three Outputs from the Same Question ```js // roles.mjs — the same question, the same table, three job descriptions: three separate outputs side by side. // Question: "is the subscriber count falling across the network, and what should be done." import { point } from "./series.mjs"; const P0 = point(0), P30 = point(30), P90 = point(90); const pct = (next, prev) => ((100 * (next - prev)) / prev).toFixed(2); const THRESHOLD = -5.0, HORIZON = 365; console.log("role 1 measuring — does not change the question; delivers the number and its decisions"); console.log(` subscribers over 90 days ${P90.n} -> ${P0.n}: ${P0.n - P90.n} subscribers, ${pct(P0.n, P90.n)}%`); console.log(" decisions: open contract definition, comparison period 90 days, no breakdown (3)"); console.log("\nrole 2 decision-binding — turns the criterion into fee and ties it to a threshold"); console.log(` fee over 90 days ${P90.u} -> ${P0.u}: ${P0.u - P90.u}, ${pct(P0.u, P90.u)}%`); const change = Number(pct(P0.u, P90.u)); console.log(` threshold ${THRESHOLD.toFixed(2)}%: ${change <= THRESHOLD ? "exceeded, action needed" : "not exceeded, no action"}`); console.log(` fee per subscriber ${P90.per.toFixed(2)} -> ${P0.per.toFixed(2)}: ` + `${pct(P0.per, P90.per)}%`); console.log(" decisions: criterion fee, threshold -5%, period 90 days, no breakdown (4)"); console.log("\nrole 3 generalizing — carries the question into the future; does not deliver a single number"); const extend = (p) => Math.round(P0.n + ((P0.n - p.n) / p.d) * HORIZON); const a = extend(P30), b = extend(P90); console.log(` at the pace of the last 30 days, after ${HORIZON} days: ${a} subscribers`); console.log(` at the pace of the last 90 days, after ${HORIZON} days: ${b} subscribers`); console.log(` difference between two defensible windows: ${Math.abs(a - b)} subscribers`); console.log(" decisions: forecast window, linear drift assumption, no seasonal data (3)"); console.log(`\nthree correct numbers from the same table: subscribers ${pct(P0.n, P90.n)}%, ` + `fee ${pct(P0.u, P90.u)}%, fee per subscriber ${pct(P0.per, P90.per)}%`); console.log("all three come from the same rows; two show a drop, one shows a rise"); ``` ``` role 1 measuring — does not change the question; delivers the number and its decisions subscribers over 90 days 4439 -> 4211: -228 subscribers, -5.14% decisions: open contract definition, comparison period 90 days, no breakdown (3) role 2 decision-binding — turns the criterion into fee and ties it to a threshold fee over 90 days 139508 -> 134843: -4665, -3.34% threshold -5.00%: not exceeded, no action fee per subscriber 31.43 -> 32.02: 1.89% decisions: criterion fee, threshold -5%, period 90 days, no breakdown (4) role 3 generalizing — carries the question into the future; does not deliver a single number at the pace of the last 30 days, after 365 days: 2191 subscribers at the pace of the last 90 days, after 365 days: 3286 subscribers difference between two defensible windows: 1095 subscribers decisions: forecast window, linear drift assumption, no seasonal data (3) three correct numbers from the same table: subscribers -5.14%, fee -3.34%, fee per subscriber 1.89% all three come from the same rows; two show a drop, one shows a rise ``` The first role answers the question as it stands: in 90 days, 228 subscribers were lost, 5.14%. Three decisions come with it. There is no judgment inside this output: it does not say whether 5.14% is bad, because saying so requires a written threshold, and the threshold is outside this job description. The second role changes the criterion and writes down its reason: a subscriber count cannot be tied to an action, because what the lost subscribers carried is not visible in the count. With the fee criterion the drop is 3.34%, which falls short of the -5% threshold; the decision is "no action." The third line writes the same movement with the opposite sign: fee per subscriber has **risen** 1.89%. Why three numbers from the same rows go in different directions will be printed in the next section. There is a danger here, and it should be named. Changing the criterion and landing below the threshold differs from disliking a number and looking for another one in only one respect: **if the threshold was written before the number, it is a decision; if it was written after, it is a justification.** AC21 wrote the threshold before the calculation; that is why the output of the second role is defensible. If the same threshold had been pulled to -3% after the result was seen, the calculation would stay the same and the output would stop being a decision. The third role does not deliver a single number, and this is not a shortcoming but a requirement of the job description. Two defensible forecasts come out of the same series: 2,191 subscribers at the pace of the last 30 days, 3,286 at the pace of the last 90. The difference is 1,095 subscribers, a quarter of today's count. To choose between the two one would need to know whether the drop is seasonal or ongoing, and there is no history at hand to separate them. The depth requirement of the second lesson comes back here as a question: four periods of history do not show an annual wave. The real output of the third role is therefore not a number but a **data request**. #### The Breakdown None of the Three Wrote The decision lists of the three roles share a line: **no breakdown.** All three worked on the network total, because the acceptance criterion of each was satisfied by the total. Yet why the three numbers diverge from one another is written in a single breakdown. ```js // breakdown.mjs — the place where all three roles wrote "no breakdown": how the loss spreads across tariff tiers. import { openAt } from "./series.mjs"; const T0 = openAt(0), T90 = openAt(90); const count = (l, b) => l.filter((k) => k.tier === b).length; const feeSum = (l, b) => l.filter((k) => k.tier === b).reduce((s, k) => s + k.fee, 0); const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const y = gen([10, 10, 10, 9, 12, 12, 10]); console.log(y(["tier", "90 days", "today", "change", "fee 90", "fee today", "fee chg."])); for (const b of [1, 2, 3]) { console.log(y([b, count(T90, b), count(T0, b), count(T0, b) - count(T90, b), feeSum(T90, b), feeSum(T0, b), feeSum(T0, b) - feeSum(T90, b)])); } const net = T0.length - T90.length; const t1 = count(T0, 1) - count(T90, 1); console.log(`\nof the ${net} subscription change, ${t1} is in tier 1: ` + `${((100 * t1) / net).toFixed(1)}%`); console.log(`tier 1 carries ${(feeSum(T90, 1) / count(T90, 1)).toFixed(2)} fee per subscriber, ` + `tier 3 ${(feeSum(T90, 3) / count(T90, 3)).toFixed(2)}`); ``` ``` tier 90 days today change fee 90 fee today fee chg. 1 2058 1899 -159 26758 24706 -2052 2 1482 1420 -62 50512 48430 -2082 3 899 892 -7 62238 61707 -531 of the -228 subscription change, -159 is in tier 1: 69.7% tier 1 carries 13.00 fee per subscriber, tier 3 69.23 ``` 69.7% of the loss is in the first tariff tier. A subscriber in the first tier carries 13.00 fee units, one in the third tier 69.23, more than five times as much. These two numbers explain all three at once: the subscriber count drops sharply because the loss is in the most crowded tier, total fee drops more slowly because that tier carries the least fee, and fee per subscriber rises because the rows pulling the average down are leaving. The breakdown itself is also a decision, and breaking down by tariff tier was not the only option; breaking down by region, by subscriber age, or by consumption level would have given a different table. But none of the three roles wrote any breakdown at all. **Whatever an acceptance criterion makes mandatory to deliver is produced; whatever it does not make mandatory is not produced, even when it is needed.** The line that would have made the three correct numbers readable together was inside the job description of none of them. #### When an Output Is Read in the Wrong Role All three outputs are correct, but when one is read in place of another, a wrong sentence appears. If the first role's "-228 subscribers" output is read as a decision, the reader fills in the missing threshold from their own head, and a limit that nobody wrote ends up in the report. If the second role's "threshold not exceeded" output is read as a measurement, the information that 228 subscribers left is lost: that sentence does not measure, it compares against a threshold. If the third role's "3,286" is read as a measurement, the linear drift assumption begins to travel as a fact, whereas the same series gives 2,191 with a different window. The total count says something too. Behind a one-sentence question stand **10 decisions** written by the three roles. None of them changes the data, and all of them change the result. If one of the three outputs sits in a report, the job description that produced it and the decisions that were written should sit next to it. Otherwise the rule of the course applies: a number whose decisions are not written counts as unmeasured. #### Summary - Three job descriptions are separated by their acceptance criteria: the measuring role delivers the number and its decisions, the decision-binding role delivers the threshold the number is tied to, and the generalizing role delivers a generalization together with its assumption. - Three correct numbers come from the same table: subscribers -5.14%, total fee -3.34%, fee per subscriber 1.89%. Two show a drop and one shows a rise. - Changing the criterion is a decision if the threshold was written before the calculation and a justification if it was written after; the only thing that makes the distinction is the order. - The generalizing role's two defensible forecasts lie between 2,191 and 3,286, a gap of 1,095 subscribers; since there is no history at hand to make the choice, the output is not a number but a data request. - Why the three numbers go in different directions is written in a single breakdown: 69.7% of the loss is in the lowest tariff tier, and the acceptance criterion of none of the three roles makes a breakdown mandatory. - Behind a one-sentence question stand 10 written decisions; an output cannot be reported without writing which job description it came from. #### Next Step Throughout this topic, four lessons did the same thing: they made the question measurable, changed the definition, and wrote down how much the number moved. The table itself was always taken as given. All the decisions taken were on the side of the question: which column is the criterion, which window, which threshold, which period. In the last block of this lesson an operation was performed and never questioned: the average of the fee field was taken and reported as "fee per subscriber." The same calculation could have been applied to the tariff tier field, and it would have given a number, the average of the tiers. That number can also be calculated, written down, and put in a table, and it would mean nothing. What decides the difference between the two is not the calculation but **what kind of thing** the field is. Contract status is a class name, tariff tier is an ordered level, fee is a measured quantity, and subscriber ID is a label that looks like a number. The next topic builds this distinction and shows the number that appears when the type of a field is assigned wrongly. ## The Structure of Data Counting what kind of thing the data being worked on is and which summaries that makes legitimate: the meaningless average produced by treating a discrete field as continuous, the summaries that nominal, ordinal, interval, and ratio scales permit, a primary and a secondary source giving different numbers for the same quantity, the number of rows eliminated by accuracy, completeness, consistency, and timeliness thresholds, the question that should not be asked even though the data can answer it, and the breakdown that descends to a single person. ### Data Types Source: https://academia.sh/en/courses/introduction-to-data-analytics/data-types Eight fields of 1,400 subscriber records are stored as numbers and all eight get an average, but three of them are qualitative: the average of the region code comes out 3.19 under the register coding and 2.69 under the size coding, and not a single record in the data has changed. No single record equals the 1.20 average of the discrete visit field, and 68.3% of the records fall below that average; when the continuous consumption field is rounded to 5 m³ steps the total shifts by only 1.4 per thousand, yet 229 records change tariff tier. The previous topic showed that making a question measurable is the real decision: when the definition of "active subscriber" changed, the number moved without a single calculation changing. But across those four lessons one thing was never asked — what kind of thing the **data itself** is, the data the definition works on. Can a field be averaged, and if it is, what does the result mean; is the difference between two of its values a magnitude or only a difference of labels. This question looks like a technical detail and is not one. In a dataset every field can sit there as a number; a calculator will return the average of all of them and produce an error message on none. A field with the wrong type assigned shows itself not with an error but with a **number that means nothing**. This lesson measures that number in three different forms. - **SD1.** The dataset is the subscriber records of the regional measurement network: 1,400 rows, eight fields, five regions. The set and the numbers are **fictional**; every value below is the output of a model. The same set is used in all the lessons of this topic. - **SD2. Quantitative data** reports a magnitude: the difference between two values is an amount. **Qualitative data** reports a category; the values are distinct but the difference between them is not an amount. - **SD3.** If a quantitative field is **discrete**, it takes only countable values and the in-between values have no counterpart; if it is **continuous**, there is always another value between any two, and measurement is cut off at some precision. - **SD4.** A field's **storage type** and its **data type** are not the same thing. A field stored as an integer can be qualitative; a field stored with decimals can in fact be discrete. - **SD5.** The tariff tier thresholds are 10, 25, and 40 m³. These thresholds are **data**; on this page they are not measured but given to the model as input. #### Storage Type and Data Type Are Not the Same Thing ```js // dataset.mjs — regional measurement network record: each row is a subscriber, its fields // are a meter reading, a field visit, and a satisfaction score. A MODEL, the dataset is fictional. export const SEED = 20260218, RECORD_COUNT = 1400; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; export const REGION = [ { name: "north", code: 4, share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", code: 3, share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", code: 2, share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", code: 1, share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", code: 5, share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; export const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const consumption = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); return { subscriber_id: 10001 + i, region: b.name, consumption_m3: Math.round(consumption * 100) / 100, meter_count: pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1, visit_count: pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]), satisfaction: pick(r(), b.sat) + 1, temperature_c: Math.round((6 + r() * 22) * 10) / 10, days_elapsed: Math.floor(Math.pow(r(), 1.9) * 210) }; }); // Coding A: the region codes fixed in the subscriber register (each REGION row carries its code). export const CODE_A = Object.fromEntries(REGION.map((b) => [b.name, b.code])); export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length; const distinct = (f) => new Set(RECORDS.map(f)).size; export const FIELDS = [ { name: "subscriber_id", stored: "integer", kind: "qualitative", measure: "-", f: (k) => k.subscriber_id }, { name: "region_code", stored: "integer", kind: "qualitative", measure: "-", f: (k) => CODE_A[k.region] }, { name: "satisfaction", stored: "integer", kind: "qualitative", measure: "ordinal", f: (k) => k.satisfaction }, { name: "meter_count", stored: "integer", kind: "quantitative", measure: "discrete", f: (k) => k.meter_count }, { name: "visit_count", stored: "integer", kind: "quantitative", measure: "discrete", f: (k) => k.visit_count }, { name: "days_elapsed", stored: "integer", kind: "quantitative", measure: "discrete", f: (k) => k.days_elapsed }, { name: "consumption_m3", stored: "decimal", kind: "quantitative", measure: "continuous", f: (k) => k.consumption_m3 }, { name: "temperature_c", stored: "decimal", kind: "quantitative", measure: "continuous", f: (k) => k.temperature_c }, ]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model (seed ${SEED}): ${RECORD_COUNT} subscriber records, ${REGION.length} regions, ` + `${FIELDS.length} fields; all eight fields are stored as numbers`); const y = gen([16, 11, 14, 12, 10, 12]); console.log("\n" + y(["field", "stored", "type", "measure", "distinct", "mean"])); for (const a of FIELDS) { console.log(y([a.name, a.stored, a.kind, a.measure, distinct(a.f), mean(RECORDS.map(a.f)).toFixed(2)])); } const meaningful = FIELDS.filter((a) => a.kind === "quantitative").length; console.log(`\nmeans were computed for all eight fields; ` + `${meaningful} are quantitative, ${FIELDS.length - meaningful} are not`); } ``` ``` model (seed 20260218): 1400 subscriber records, 5 regions, 8 fields; all eight fields are stored as numbers field stored type measure distinct mean subscriber_id integer qualitative - 1400 10700.50 region_code integer qualitative - 5 3.19 satisfaction integer qualitative ordinal 5 3.56 meter_count integer quantitative discrete 4 1.55 visit_count integer quantitative discrete 6 1.20 days_elapsed integer quantitative discrete 209 69.39 consumption_m3 decimal quantitative continuous 1099 21.31 temperature_c decimal quantitative continuous 221 16.98 means were computed for all eight fields; 5 are quantitative, 3 are not ``` The last column is the real finding of this lesson: an average came out for all eight fields. The average of the `subscriber_id` field is 10,700.50, and this number is correct — it really is the arithmetic mean of the 1,400 subscriber numbers. The same number has no counterpart at all: there is no such concept as an average subscriber number, because the difference between subscriber numbers is not an amount but a distinction. The number was computed, computed correctly, and **there is nothing being measured**. The `distinct` column is a hint for guessing the type but does not decide it. The `region_code` field takes five distinct values, and from this one can sense that it is qualitative. `meter_count` also takes only four values, yet it is quantitative and discrete: a household with two meters really does have one meter more than a household with one. What makes the distinction is not the number of values but **whether the difference between values reports an amount**. The `consumption_m3` field takes 1,099 distinct values; this shows not that the field is continuous but that the measurement was cut off at two decimal places. A continuous quantity always looks discrete the moment it is stored; continuity is a property not of the stored value but of **the thing measured**. The `satisfaction` field is in a third situation. Its five values are ordered — a score of 4 is better than a score of 3 — but no measurement says that the distance between 4 and 3 equals the distance between 2 and 1. This lesson counts it as qualitative and notes that it is ordered; which summaries ordering permits is the subject of the next lesson. #### The Number a Wrongly Typed Field Produces Type assignment is not a labeling chore; it moves the number that goes into the report directly. The block below measures three wrong assignments in turn. ```js // type-error.mjs — the two numbers produced when the same field is read as two types, side by side. import { RECORDS, REGION, CODE_A, mean } from "./dataset.mjs"; // Coding B: regions are numbered from largest to smallest by record count. const counts = Object.fromEntries(REGION.map((b) => [b.name, RECORDS.filter((k) => k.region === b.name).length])); const CODE_B = Object.fromEntries(Object.keys(counts) .sort((x, y) => counts[y] - counts[x]).map((name, j) => [name, j + 1])); const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); console.log("region_code is a qualitative field; both codings are defensible"); const y1 = gen([10, 9, 9, 9]); console.log(y1(["region", "records", "code A", "code B"])); for (const name of Object.keys(counts)) console.log(y1([name, counts[name], CODE_A[name], CODE_B[name]])); const mA = mean(RECORDS.map((k) => CODE_A[k.region])); const mB = mean(RECORDS.map((k) => CODE_B[k.region])); console.log(`code A mean ${mA.toFixed(2)}, code B mean ${mB.toFixed(2)}, ` + `difference ${(mB - mA).toFixed(2)} — not a single record changed in the data`); // Discrete field: the mean and the distribution side by side. const z = RECORDS.map((k) => k.visit_count), mz = mean(z); const VALUES = [0, 1, 2, 3, 4, 5]; const count = (v) => z.filter((x) => x === v).length; // Decimal equality is not tested directly; the 0.005 threshold is half a percent of one visit // and in this set can never count two different integer values as the same. const EPS = 0.005, equal = z.filter((v) => Math.abs(v - mz) < EPS).length; const below = z.filter((v) => v < mz).length; console.log(`\nvisit_count: mean ${mz.toFixed(2)}, median ` + `${[...z].sort((a, b) => a - b)[z.length / 2]}, mode ` + `${VALUES.reduce((a, v) => (count(v) > count(a) ? v : a), 0)}`); const y2 = gen([10, 9, 10, 12]); console.log(y2(["visits", "records", "percent", "cumulative"])); let cum = 0; for (const v of VALUES) { cum += count(v); console.log(y2([v, count(v), (100 * count(v) / z.length).toFixed(1), cum])); } console.log(`records equal to the mean ${equal}; records below the mean ${below}, ` + `${(100 * below / z.length).toFixed(1)}%`); // Continuous field read as discrete: rounding to 5 m3 steps. const THRESHOLDS = [10, 25, 40]; const tier = (v) => (v < THRESHOLDS[0] ? 1 : v < THRESHOLDS[1] ? 2 : v < THRESHOLDS[2] ? 3 : 4); const raw = RECORDS.map((k) => k.consumption_m3), rounded = raw.map((v) => Math.round(v / 5) * 5); const shifted = raw.filter((v, i) => tier(v) !== tier(rounded[i])).length; const sum = (a) => a.reduce((x, y) => x + y, 0); console.log(`\nconsumption_m3 is continuous; rounded to 5 m3 steps the total becomes ` + `${sum(rounded).toFixed(1)} m3 instead of ${sum(raw).toFixed(1)} m3, a difference of ` + `${(sum(rounded) - sum(raw)).toFixed(1)} m3 (${(1000 * (sum(rounded) - sum(raw)) / sum(raw)).toFixed(1)} per thousand)`); console.log(`mean ${mean(rounded).toFixed(2)} instead of ${mean(raw).toFixed(2)}; ` + `${shifted} records change tariff tier`); const y3 = gen([12, 9, 14, 9]); console.log(y3(["tier", "raw", "rounded", "diff"])); for (const b of [1, 2, 3, 4]) { const h = raw.filter((v) => tier(v) === b).length; const u = rounded.filter((v) => tier(v) === b).length; console.log(y3([`${b} (${b === 4 ? "40+" : `<${THRESHOLDS[b - 1]}`} m3)`, h, u, u - h])); } ``` ``` region_code is a qualitative field; both codings are defensible region records code A code B north 408 4 1 south 282 3 2 east 240 2 4 west 209 1 5 central 261 5 3 code A mean 3.19, code B mean 2.69, difference -0.51 — not a single record changed in the data visit_count: mean 1.20, median 1, mode 0 visits records percent cumulative 0 514 36.7 514 1 442 31.6 956 2 237 16.9 1193 3 106 7.6 1299 4 62 4.4 1361 5 39 2.8 1400 records equal to the mean 0; records below the mean 956, 68.3% consumption_m3 is continuous; rounded to 5 m3 steps the total becomes 29870.0 m3 instead of 29828.2 m3, a difference of 41.8 m3 (1.4 per thousand) mean 21.34 instead of 21.31; 229 records change tariff tier tier raw rounded diff 1 (<10 m3) 48 3 -45 2 (<25 m3) 970 853 -117 3 (<40 m3) 348 488 140 4 (40+ m3) 34 56 22 ``` The first block measures the average applied to a qualitative field. The region code can be given in two forms and both are defensible: numbering the regions with the codes fixed in the subscriber register, or numbering them from largest to smallest by record count. Under the first coding the average region code comes out 3.19, under the second 2.69. No record in the data has changed; the only thing that changed is the numbers attached to the labels. This is the shortest proof that the average of a qualitative field **belongs not to the data but to the coding**. If a report has a line saying "average region code 3.19", that line is not a finding but the shadow of an ordering decision. The second block shows what happens when a discrete field is treated as continuous. The average number of visits is 1.20; the calculation is correct. But **none** of the 1,400 records received 1.20 visits, because there is no such thing as half a visit. The number of records equal to the average is zero. Moreover, 68.3% of the records fall below this average: the largest group, at 514 records, is the subscribers who received **no visit at all**. The sentence "1.20 field visits per subscriber" does not describe even a single subscriber; the sentence "36.7% of subscribers received no visit at all" comes out of the same data and says something entirely different. Both are correct and the two suggest opposite decisions — one shows that the visit load is balanced, the other that a segment is never seen. For a discrete field, what goes into the report is not the average but **the distribution**; the average is only a summary placed next to that distribution. The third block measures a continuous field being read as discrete, and its surprising result is here. When consumption is rounded to 5 m³ steps, the total goes from 29,828.2 m³ to 29,870.0 m³: a difference of 1.4 per thousand. The total barely stirs, because the upward and downward roundings cancel each other out. The same operation changes the tariff tier of 229 records. The 48 records in the first tier drop to 3, and the third tier rises from 348 to 488. **Rounding preserves the total and corrupts the classification.** Someone reporting on total consumption can say that rounding is harmless, and is right; someone reporting on the tariff tier distribution says that the same operation moves one in six records to another tier, and is also right. Which decision is harmless depends on the question being asked. All three measurements write the same rule: a number whose type assignment is not written next to it cannot be read. Under which coding the region code average was computed, next to which distribution the visit average sits, whether the consumption average was rounded — without these three pieces of information, none of the three numbers can be defended. #### Summary - Storage type does not determine data type: all eight fields sit there as numbers, three are qualitative, and even though an average can be computed for them it does not report a magnitude. - What separates quantitative from qualitative is not the number of values but whether the difference between two values reports an amount; `meter_count` is quantitative with four values, `region_code` is qualitative with five. - The average of a qualitative field belongs to the coding: the same 1,400 records give 3.19 under the register coding and 2.69 under the size coding, and nothing in the data has changed. - When a discrete field is treated as continuous, the average falls to a value with no counterpart: no record equals the 1.20 visit average, and 68.3% of the records fall below that value. - Rounding a continuous field moves the total by 1.4 per thousand but changes the tariff tier of 229 records; whether the same decision is harmless changes with the question being asked. #### Next Step This lesson asked whether the average of a field can be taken, and was left with a half answer on the `satisfaction` field: its values are ordered, but nobody has measured that the distance between 4 and 3 equals the distance between 2 and 1. The same uncertainty exists in the `temperature_c` field too — can two temperatures be added, can one be counted as twice the other. The quantitative and qualitative distinction is not enough to answer these questions, because the distinction is too coarse. The next lesson places the fields on four **measurement scales** and counts which summary is legitimate on each scale: two defensible numeric codings given to an ordinal field reverse the satisfaction ranking of the regions from start to finish. ### Measurement Scales Source: https://academia.sh/en/courses/introduction-to-data-analytics/measurement-scales Ten fields are placed on four scales and all 40 cells formed by multiplying them with four summaries are computed: 28 turn out to be legitimate, and 12 are numbers that were computed but that the scale does not support. Two defensible codings given to the ordinal satisfaction field — 1, 2, 3, 4, 5 and 0, 4, 6, 7, 8 — reverse the ranking of the regions: the first region falls to fifth without a single grade changing. On an interval scale the ratio of the same two readings falls from 4.67 to 1.08 when the zero point changes, while their difference stays fixed at 22.0 units. The previous lesson divided the fields into quantitative and qualitative, and got stuck in two places. The values of the `satisfaction` field are ordered, but nobody has measured that the distance between 4 and 3 equals the distance between 2 and 1. The `temperature_c` field is quantitative, but whether one reading can be counted as twice another was left uncertain. Both point to the same place: the distinction between quantitative and qualitative is too coarse. This lesson places the fields on four **measurement scales**. A scale says which operation is meaningful on a field's values; and the set of meaningful operations determines which summary is legitimate. The finding of the lesson is this: that a calculation can be done does not mean it is legitimate. - **SD6.** The set is the 1,400-row subscriber record from the previous lesson; two derived fields are added: `tariff_tier` (the tier into which consumption falls according to the 10, 25, and 40 m³ thresholds) and `reading_day` (the number of the day in the observation window on which the reading was taken). The set is **fictional**. - **SD7. The nominal scale** permits only equality. **The ordinal scale** adds ordering but does not report the distance between two neighboring steps. - **SD8. The interval scale** makes differences meaningful; its zero point is chosen by convention and does not report an absence. **The ratio scale** rests on a true zero, so ratio statements are meaningful too. - **SD9.** Four summaries demand four separate operations: the mode demands equality, the median ordering, the mean equal intervals, and a ratio statement a true zero. If the operation it demands is not defined on the scale, a summary is **computable but not legitimate**. #### How Many of the Forty Cells Are Legitimate ```js // scales.mjs — the same set, this time every field is placed on a scale and four // summaries are computed for every field. A MODEL, the dataset is fictional. export const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40]; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; export const REGION = [ { name: "north", code: 4, share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", code: 3, share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", code: 2, share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", code: 1, share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", code: 5, share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; export const REGION_CODE = Object.fromEntries(REGION.map((b) => [b.name, b.code])); export const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const consumption = Math.round(t * 100) / 100; const meters = pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1; const visits = pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]); const satisfaction = pick(r(), b.sat) + 1; const temperature = Math.round((6 + r() * 22) * 10) / 10; const elapsed = Math.floor(Math.pow(r(), 1.9) * 210); return { subscriber_id: 10001 + i, region: b.name, consumption_m3: consumption, meter_count: meters, visit_count: visits, satisfaction, temperature_c: temperature, days_elapsed: elapsed, reading_day: 300 - elapsed, tariff_tier: consumption < TARIFF[0] ? 1 : consumption < TARIFF[1] ? 2 : consumption < TARIFF[2] ? 3 : 4 }; }); // The scale says which operation is meaningful on the field's values. export const SCALES = { nominal: { equal: 1, order: 0, diff: 0, ratio: 0 }, ordinal: { equal: 1, order: 1, diff: 0, ratio: 0 }, interval: { equal: 1, order: 1, diff: 1, ratio: 0 }, ratio: { equal: 1, order: 1, diff: 1, ratio: 1 }, }; export const FIELDS = [ { name: "subscriber_id", scale: "nominal", f: (k) => k.subscriber_id }, { name: "region (code)", scale: "nominal", f: (k) => REGION_CODE[k.region] }, { name: "satisfaction", scale: "ordinal", f: (k) => k.satisfaction }, { name: "tariff_tier", scale: "ordinal", f: (k) => k.tariff_tier }, { name: "temperature_c", scale: "interval", f: (k) => k.temperature_c }, { name: "reading_day", scale: "interval", f: (k) => k.reading_day }, { name: "consumption_m3", scale: "ratio", f: (k) => k.consumption_m3 }, { name: "meter_count", scale: "ratio", f: (k) => k.meter_count }, { name: "visit_count", scale: "ratio", f: (k) => k.visit_count }, { name: "days_elapsed", scale: "ratio", f: (k) => k.days_elapsed }, ]; export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length; const mode = (a) => { const s = new Map(); for (const v of a) s.set(v, (s.get(v) ?? 0) + 1); return [...s].sort((x, y) => y[1] - x[1] || x[0] - y[0])[0][0]; }; const median = (a) => [...a].sort((x, y) => x - y)[Math.floor(a.length / 2)]; const regionRatio = (f) => { const o = REGION.map((b) => mean(RECORDS.filter((k) => k.region === b.name).map(f))); return Math.max(...o) / Math.min(...o); }; // Each summary demands one operation: the mode demands equality, the median ordering, // the arithmetic mean equal intervals, a ratio statement a true zero point. export const SUMMARIES = [ { name: "mode", demands: "equal", h: (f) => mode(RECORDS.map(f)).toFixed(1) }, { name: "median", demands: "order", h: (f) => median(RECORDS.map(f)).toFixed(1) }, { name: "mean", demands: "diff", h: (f) => mean(RECORDS.map(f)).toFixed(2) }, { name: "region ratio", demands: "ratio", h: (f) => regionRatio(f).toFixed(2) }, ]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model (seed ${SEED}): ${RECORD_COUNT} records, ${FIELDS.length} fields, ` + `${SUMMARIES.length} summaries — all ${FIELDS.length * SUMMARIES.length} cells can be computed`); const y = gen([17, 14, 12, 11, 11, 14, 12]); console.log("\n" + y(["field", "scale", ...SUMMARIES.map((o) => o.name), "legitimate"])); let legit = 0; for (const a of FIELDS) { const allowed = SCALES[a.scale]; const cells = SUMMARIES.map((o) => o.h(a.f) + (allowed[o.demands] ? " " : "*")); const m = SUMMARIES.filter((o) => allowed[o.demands]).length; legit += m; console.log(y([a.name, a.scale, ...cells, `${m}/${SUMMARIES.length}`])); } console.log(`\n* marked cell was computed but the scale does not support that operation`); console.log(`${legit} of ${FIELDS.length * SUMMARIES.length} cells are legitimate, ` + `${FIELDS.length * SUMMARIES.length - legit} are not`); } ``` ``` model (seed 20260218): 1400 records, 10 fields, 4 summaries — all 40 cells can be computed field scale mode median mean region ratio legitimate subscriber_id nominal 10001.0 10701.0* 10700.50* 1.00* 1/4 region (code) nominal 4.0 3.0* 3.19* 5.00* 1/4 satisfaction ordinal 4.0 4.0 3.56* 1.08* 2/4 tariff_tier ordinal 2.0 2.0 2.26* 1.38* 2/4 temperature_c interval 7.7 16.8 16.98 1.03* 3/4 reading_day interval 300.0 249.0 230.61 1.03* 3/4 consumption_m3 ratio 14.5 20.0 21.31 1.86 4/4 meter_count ratio 1.0 1.0 1.55 1.09 4/4 visit_count ratio 0.0 1.0 1.20 1.11 4/4 days_elapsed ratio 0.0 51.0 69.39 1.10 4/4 * marked cell was computed but the scale does not support that operation 28 of 40 cells are legitimate, 12 are not ``` All forty of the forty cells gave a number and none produced an error. The legitimate ones are 28; 12 cells used an operation the scale does not support. These 12 numbers were not computed wrongly — the region ratio of the `region (code)` field really is 5.00. That ratio has no counterpart: once region codes are ordered, the fifth region does not have something "five times" more than the first. As the scale rises the number of legitimate summaries grows, from 1 on the nominal scale to 4 on the ratio scale; the scales contain one another. This is why **placing a field on a higher scale than it has is a silent error**: the calculation is done, a number comes out, no warning arrives. Read in the opposite direction, a second warning emerges. Being legitimate is not being useful. The mode of the `consumption_m3` field is 14.5 and is legitimate on the ratio scale; but because the field takes 1,099 distinct values, 14.5 is a reading that appears only five times, and there is a second value that ties with it at the same five. The mode does not describe the set; the scale sets a lower bound, not an upper bound. #### Whom the Average of an Ordinal Field Belongs To The most dangerous of the starred cells is the 3.56 average of the `satisfaction` field, because it does not look wrong. The block below measures what that number depends on. ```js // coding.mjs — the average of an ordinal field under two defensible codings side by side; // also the dependence of a ratio statement on the zero point on an interval scale. import { RECORDS, REGION, mean } from "./scales.mjs"; // A: the steps are equally spaced. B: the distance at the ends is larger — // "very bad" is farther from "bad" than "good" is from "very good". const CODE_A = [1, 2, 3, 4, 5], CODE_B = [0, 4, 6, 7, 8]; const GRADE = ["very bad", "bad", "fair", "good", "very good"]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const median = (a) => [...a].sort((x, y) => x - y)[Math.floor(a.length / 2)]; console.log("satisfaction steps and the two codings"); const y0 = gen([12, 11, 8, 8]); console.log(y0(["step", "grade", "code A", "code B"])); GRADE.forEach((n, j) => console.log(y0([j + 1, n, CODE_A[j], CODE_B[j]]))); const rows = REGION.map((b) => { const l = RECORDS.filter((k) => k.region === b.name).map((k) => k.satisfaction); return { name: b.name, n: l.length, distribution: [1, 2, 3, 4, 5].map((v) => l.filter((x) => x === v).length), a: mean(l.map((v) => CODE_A[v - 1])), b: mean(l.map((v) => CODE_B[v - 1])), med: median(l) }; }); const rank = (key) => { const s = [...rows].sort((x, y) => y[key] - x[key]); return Object.fromEntries(s.map((x, j) => [x.name, j + 1])); }; const rA = rank("a"), rB = rank("b"); console.log("\nsatisfaction per region: the distribution is the same, the two codings give two rankings"); const y1 = gen([9, 9, 22, 9, 8, 9, 8, 9]); console.log(y1(["region", "records", "distribution 1..5", "code A", "rank A", "code B", "rank B", "median"])); for (const s of rows) { console.log(y1([s.name, s.n, s.distribution.join(" "), s.a.toFixed(2), rA[s.name], s.b.toFixed(2), rB[s.name], s.med])); } const moved = rows.filter((s) => rA[s.name] !== rB[s.name]).length; console.log(`${moved} regions changed rank; largest shift ` + `${rows.map((s) => Math.abs(rA[s.name] - rB[s.name])).sort((x, y) => y - x)[0]} places`); const topA = rows.find((s) => rA[s.name] === 1), topB = rows.find((s) => rB[s.name] === 1); console.log(`code A ranks ${topA.name} first, code B ranks ${topB.name} first; ` + `${topA.name} is at rank ${rB[topA.name]} under code B`); console.log(`both calculations are correct, both codings are defensible, not a single grade changed`); // Interval scale: the zero point is set by convention, the ratio depends on it. const warm = RECORDS.map((k) => k.temperature_c); const SHIFT = [0, -273.15, 32]; // the value of the same measurement at three different zero points const LABEL = ["local scale", "absolute-zero scale", "another convention"]; const a = Math.max(...warm), b = Math.min(...warm); console.log(`\nhottest reading ${a.toFixed(1)}, coldest reading ${b.toFixed(1)} (local scale)`); const y2 = gen([24, 11, 11, 9, 9]); console.log(y2(["scale", "hottest", "coldest", "diff", "ratio"])); SHIFT.forEach((d, j) => { const x = j === 2 ? a * 1.8 + 32 : a - d, y = j === 2 ? b * 1.8 + 32 : b - d; console.log(y2([LABEL[j], x.toFixed(1), y.toFixed(1), (x - y).toFixed(1), (x / y).toFixed(2)])); }); console.log(`the first two scales share a unit but have different zero points: the difference ` + `stays fixed at ${(a - b).toFixed(1)} units, the ratio falls from 4.67 to 1.08`); ``` ``` satisfaction steps and the two codings step grade code A code B 1 very bad 1 0 2 bad 2 4 3 fair 3 6 4 good 4 7 5 very good 5 8 satisfaction per region: the distribution is the same, the two codings give two rankings region records distribution 1..5 code A rank A code B rank B median north 408 17 37 107 170 77 3.62 2 6.36 2 4 south 282 8 23 89 123 39 3.57 3 6.38 1 4 east 240 44 25 17 24 130 3.71 1 5.88 5 5 west 209 16 34 47 67 45 3.44 4 5.97 4 4 central 261 18 45 55 94 49 3.43 5 5.98 3 4 3 regions changed rank; largest shift 4 places code A ranks east first, code B ranks south first; east is at rank 5 under code B both calculations are correct, both codings are defensible, not a single grade changed hottest reading 28.0, coldest reading 6.0 (local scale) scale hottest coldest diff ratio local scale 28.0 6.0 22.0 4.67 absolute-zero scale 301.1 279.1 22.0 1.08 another convention 82.4 42.8 39.6 1.93 the first two scales share a unit but have different zero points: the difference stays fixed at 22.0 units, the ratio falls from 4.67 to 1.08 ``` Both codings are defensible. The first counts the steps as equally spaced. The second takes the gap between "very bad" and "bad" as farther than the other neighboring steps: the lowest step reports a complaint, and its difference from the one above is larger than the difference between "good" and "very good". Which one is correct **the data does not say**, because nothing in the data measures the distance between steps. The result is that two different rankings come out of the same 1,400 grades. Under code A the first region is `east`, under code B it is fifth; the rank of three regions changed, and the largest shift is four places. The cause is written in the distribution column: of the 240 records of the `east` region, 130 are on the top step and 44 on the bottom. This polarized distribution rises under the coding that counts the steps as equal, and sinks to the bottom under the one that penalizes the lower end heavily. The 408 records of the `north` region are gathered in the middle, and its rank does not change under either coding. **Two correct numbers contradicting each other came out of the same data.** The sentences "the region with the highest satisfaction is `east`" and "the region with the lowest satisfaction is `east`" were both produced from the same grades, without any arithmetic error. The only difference between them is a coding decision, and that decision appears in no report. The median column points to a way out, but not a complete solution. The median gives the same step under any increasing coding, because it relies on nothing but ordering. Its price is resolution: four of the five regions have a median of 4. The decision rule follows from this: if a ranking is needed, **the distribution itself is reported**; if an average is to be given, how it was produced, which coding, is written next to it. The last table gives the counterpart on an interval scale. The ratio of the hottest and the coldest reading is 4.67 on the local scale; on a scale whose zero point is 273.15 units lower, the ratio of the same two readings falls to 1.08, while their difference is 22.0 units on both scales. For a ratio to belong to the data, zero has to report an **absence**. In the `consumption_m3` field zero consumption really is an absence of consumption; the statement that one region consumed 1.86 times as much as another therefore comes from the data, not from the scale. #### Summary - The four scales contain one another: nominal equality, ordinal ordering, interval difference, ratio division. - All forty of the forty cells were computed and 28 turned out legitimate; 12 cells are numbers resting on an operation the scale does not support, and they produce no error. - Being legitimate is not being useful: the mode of the `consumption_m3` field is legitimate on the ratio scale but is a reading that appears five times among 1,099 distinct values and ties with another value. - The average of an ordinal field belongs to the coding: two defensible codings reverse the ranking of the regions, and the first region falls to fifth. - On an interval scale the difference belongs to the data and the ratio to the zero point: the ratio of the same two readings is 4.67, and when zero is shifted it becomes 1.08, while their difference stays at 22.0 units. #### Next Step These two lessons asked about the type and scale of the fields, but both took the set as given. What was not asked is **where the rows came from**. The average consumption comes out one way in records compiled from field visits, another in a summary derived from billing records, another in readings the meters send on their own. All three were collected correctly, because each source carries a decision about who gets into the record. The next lesson separates these three sources and counts how much the same quantity shifts from source to source. ### Data Sources Source: https://academia.sh/en/courses/introduction-to-data-analytics/data-sources The same quantity — average monthly consumption — gives three different numbers from three sources: 22.42 m³ from the field survey, 21.59 m³ for the value rebuilt from the billing summary, 17.01 m³ from meter telemetry, while the model's true value is 21.31 m³. Even the subscriber count comes out 1,400, 886, and 1,041 depending on the source. When a single decision is changed in each source, the six numbers spread between 14.11 and 22.42 m³ and none of them contains a calculation error; the source of the bias is visible in the coverage table. The previous two lessons asked about the type and scale of the fields, but both took the 1,400 rows as given. This lesson asks where the rows came from, and measures a single quantity — one month's average consumption — from three separate sources. All three numbers were collected correctly, and all three are different. The difference does not come from a calculation error. Each source carries a decision about who gets into the record, and that decision does not arrive with the data; it is stored in the way the source was built. A survey contains only those who responded, a telemetry stream only the devices that can transmit, a summary report only the published breakdown. The measure at the end of this lesson is **the bias a source carries**: the difference between the number the source gives and the truth, and which excluded records that difference comes from. - **SD10.** The set is the 1,400-row subscriber record of the previous lessons. The set is **fictional**, and the privilege of this lesson is the following: the model's true value is known. The true average consumption is 21.31 m³, and no source sees this number. - **SD11. Primary data** is data the party doing the analysis collected itself: the household survey filled in during a field visit. Its coverage is determined by two decisions — who was visited and who responded. - **SD12. Secondary data** is data someone else collected for another purpose and published: the billing system's tariff tier summary. There are no raw rows, only tier counts. - **SD13. Generated data** is data that accumulates by itself during the operation of a mechanism: the telemetry the meters send. Nobody fills it in, but a device failing to transmit is also a recording decision. - **SD14.** In the model, the probability of responding **rises** with consumption, the probability of transmitting **falls** with consumption, and the telemetry sends between 20 and 30 days of data each month. These three tendencies are given to the model as input; they represent the structure of the source. #### Three Sources, Three Numbers ```js // sources.mjs — a model that measures the same quantity from three sources. export const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40]; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; export const REGION = [ { name: "north", share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; export const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const consumption = Math.round(t * 100) / 100; const meters = pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1; const visits = pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]); const satisfaction = pick(r(), b.sat) + 1; const temperature = Math.round((6 + r() * 22) * 10) / 10; const elapsed = Math.floor(Math.pow(r(), 1.9) * 210); // The source draws are made AFTER the base fields; the base set does not change. const responded = r() < 0.30 + 0.014 * consumption; // heavy consumers respond more const transmitted = r() < 0.95 - 0.010 * consumption; // transmission drops more at high flow const coveredDays = 30 - Math.floor(r() * 11); // days the device reported within the month return { subscriber_id: 10001 + i, region: b.name, consumption_m3: consumption, meter_count: meters, visit_count: visits, satisfaction, temperature_c: temperature, days_elapsed: elapsed, responded, transmitted, coveredDays, tier: consumption < TARIFF[0] ? 1 : consumption < TARIFF[1] ? 2 : consumption < TARIFF[2] ? 3 : 4 }; }); export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length; const TRUTH = mean(RECORDS.map((k) => k.consumption_m3)); // Primary: the household survey collected during the field visit (those visited and those who responded). export const primary = RECORDS.filter((k) => k.visit_count >= 1 && k.responded); // Secondary: the mean rebuilt from the billing summary using tier counts. export const MIDPOINT = [5, 17.5, 32.5, 50]; export const secondaryMean = (represent) => { let t = 0; for (let b = 1; b <= 4; b++) t += RECORDS.filter((k) => k.tier === b).length * represent[b - 1]; return t / RECORD_COUNT; }; // Generated: meter telemetry; only the devices that transmit, missing days as they are. export const generated = RECORDS.filter((k) => k.transmitted); const rawTelemetry = (k) => k.consumption_m3 * k.coveredDays / 30; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model (seed ${SEED}): ${RECORD_COUNT} subscribers; true value of the model: ` + `mean ${TRUTH.toFixed(2)} m3`); const y = gen([28, 9, 10, 12, 11]); console.log("\n" + y(["source", "records", "coverage", "mean", "deviation"])); const lines = [ ["primary: field survey", primary.length, mean(primary.map((k) => k.consumption_m3))], ["secondary: billing summary", RECORD_COUNT, secondaryMean(MIDPOINT)], ["generated: meter telemetry", generated.length, mean(generated.map(rawTelemetry))], ]; for (const [name, n, o] of lines) { console.log(y([name, n, `${(100 * n / RECORD_COUNT).toFixed(1)}%`, o.toFixed(2), (o - TRUTH >= 0 ? "+" : "") + (o - TRUTH).toFixed(2)])); } console.log(`\nsubscriber count also varies by source: ${RECORD_COUNT} in the record system, ` + `${RECORDS.filter((k) => k.visit_count >= 1).length} in the field list, ` + `${generated.length} devices in telemetry`); const three = lines.map((s) => s[2]); console.log(`the largest gap among the three sources is ${(Math.max(...three) - Math.min(...three)).toFixed(2)} m3`); } ``` ``` model (seed 20260218): 1400 subscribers; true value of the model: mean 21.31 m3 source records coverage mean deviation primary: field survey 544 38.9% 22.42 +1.11 secondary: billing summary 1400 100.0% 21.59 +0.28 generated: meter telemetry 1041 74.4% 17.01 -4.30 subscriber count also varies by source: 1400 in the record system, 886 in the field list, 1041 devices in telemetry the largest gap among the three sources is 5.41 m3 ``` All three sources are built to cover the same 1,400 subscribers, and none of them does. The field survey reaches 544 records, the telemetry 1,041 devices; the billing summary covers all of them in count but shows no raw rows. The deviation column shows three different directions: the survey 1.11 m³ up, the summary 0.28 m³ up, the telemetry 4.30 m³ down. The largest gap is 5.41 m³, and this is close to a quarter of the true value. The last line is more unsettling. **Even the subscriber count changes with the source**: 1,400 in the record system, 886 in the field list, 1,041 in telemetry. The question "how many subscribers do we have" has no single answer; each number is the answer to a coverage definition. If a report states a subscriber count without saying which source was counted, that number is one of three different quantities and which one cannot be read. The 4.30 m³ deviation of the telemetry comes from two separate causes, and both belong to the structure of the source. Devices report not the whole month but between 20 and 30 days, so the raw total falls below a month's consumption. In addition, the probability of transmitting falls at high consumption, which means the devices that are missing were not chosen at random. When a source delivers **missing data**, the real question is not how much is missing but **which ones are missing**. #### When the Decision Changes Coverage is the source's decision. There is also a decision belonging to the person doing the analysis: how to extract the number from the records at hand. The block below changes a single decision in each source. ```js // decision.mjs — a single decision is changed in each source, two numbers printed side by side. import { RECORDS, RECORD_COUNT, mean, primary, generated, secondaryMean, MIDPOINT } from "./sources.mjs"; const TRUTH = mean(RECORDS.map((k) => k.consumption_m3)); const share = (b) => RECORDS.filter((k) => k.tier === b).length / RECORD_COUNT; // The tier shares are known from the billing summary; the sample is reweighted back to these shares. const weighted = (l) => { let t = 0; for (let b = 1; b <= 4; b++) { const sub = l.filter((k) => k.tier === b); if (sub.length) t += share(b) * mean(sub.map((k) => k.consumption_m3)); } return t; }; const lines = [ ["primary", "A: raw mean of respondents", mean(primary.map((k) => k.consumption_m3))], ["primary", "B: reweighting to tier shares", weighted(primary)], ["secondary", "A: each tier at its midpoint", secondaryMean(MIDPOINT)], ["secondary", "B: each tier at its lower bound", secondaryMean([0, 10, 25, 40])], ["generated", "A: missing days summed as they are", mean(generated.map((k) => k.consumption_m3 * k.coveredDays / 30))], ["generated", "B: scaled to the month by covered days", mean(generated.map((k) => k.consumption_m3))], ]; const y = (h) => String(h[0]).padEnd(11) + String(h[1]).padEnd(41) + String(h[2]).padStart(9) + String(h[3]).padStart(11); console.log(y(["source", "decision", "number", "deviation"])); for (const [k, a, s] of lines) { console.log(y([k, a, s.toFixed(2), (s - TRUTH >= 0 ? "+" : "") + (s - TRUTH).toFixed(2)])); } const s = lines.map((x) => x[2]); console.log(`\ntrue value of the model ${TRUTH.toFixed(2)} m3; none of the six numbers has a ` + `calculation error`); console.log(`largest difference within a source ${Math.max(...[0, 2, 4].map((i) => Math.abs(s[i + 1] - s[i]))).toFixed(2)} m3, range of the six numbers ` + `${Math.min(...s).toFixed(2)} - ${Math.max(...s).toFixed(2)} m3`); console.log(`mean of the A decisions of the three sources ` + `${mean([s[0], s[2], s[4]]).toFixed(2)} m3 — there is no single decision behind it`); // The source of the bias: which records fell outside the coverage. console.log(`\ncoverage tier by tier`); const y2 = (h) => String(h[0]).padEnd(10) + h.slice(1).map((v) => String(v).padStart(14)).join(""); console.log(y2(["tier", "whole set", "in survey", "survey %", "in telemetry", "telemetry %"])); for (const b of [1, 2, 3, 4]) { const t = RECORDS.filter((k) => k.tier === b).length; const a = primary.filter((k) => k.tier === b).length; const u = generated.filter((k) => k.tier === b).length; console.log(y2([b, t, a, `${(100 * a / t).toFixed(1)}%`, u, `${(100 * u / t).toFixed(1)}%`])); } ``` ``` source decision number deviation primary A: raw mean of respondents 22.42 +1.11 primary B: reweighting to tier shares 21.52 +0.21 secondary A: each tier at its midpoint 21.59 +0.28 secondary B: each tier at its lower bound 14.11 -7.19 generated A: missing days summed as they are 17.01 -4.30 generated B: scaled to the month by covered days 20.48 -0.83 true value of the model 21.31 m3; none of the six numbers has a calculation error largest difference within a source 7.48 m3, range of the six numbers 14.11 - 22.42 m3 mean of the A decisions of the three sources 20.34 m3 — there is no single decision behind it coverage tier by tier tier whole set in survey survey % in telemetry telemetry % 1 48 15 31.3% 43 89.6% 2 970 351 36.2% 754 77.7% 3 348 159 45.7% 224 64.4% 4 34 19 55.9% 20 58.8% ``` The six numbers spread between 14.11 and 22.42 m³, and all of them describe the same subscribers, the same month, the same quantity. The largest difference within a source is 7.48 m³ — larger than the difference between sources. This is the most important number of the lesson: **how you process the source moves the number as much as which source you choose**. The two decisions on the secondary source do not answer the same question, and unless this distinction is written down the two numbers cannot be compared. Using the tier midpoint is an **estimate**; using the tier lower bound is a **lower bound** and answers "at least how much do subscribers consume". The reported 14.11 m³ is not wrong, but placed next to 21.59 the reader thinks they are seeing two estimates. Whether a number is an estimate or a bound cannot be read from the number itself. On the primary source, reweighting brings the deviation down from 1.11 to 0.21 m³. The price of the correction is this: the weights come from another source, the billing summary. A second source was needed to remove the bias, and that source has its own decisions. On the telemetry, scaling by covered days brings the deviation down from 4.30 to 0.83 m³; the remaining 0.83 m³ comes from the second cause, the high consumers not transmitting, and the day scaling does not touch it. **Correcting one decision does not remove the whole bias**; the gap each decision closes is separate. The last table writes where the bias comes from. In the survey, 31.3% of the subscribers in the first tier are present, against 55.9% of those in the fourth: high consumers are overrepresented in the sample, so the mean shifts upward. In the telemetry the direction reverses — 89.6% of the first tier transmits while only 58.8% of the fourth does. Both sources are fed from the same network, in the same month, from the same subscribers, and are **biased in opposite directions**. Bias is not a property of the data but a consequence of the source's coverage decision; without the coverage table written, no deviation can be explained. Taking the average of the three sources — 20.34 m³ — is ignoring this table. There is no single decision behind that number; it is the arithmetic mean of three separate coverage decisions, and which question it answers is unclear. #### Summary - The same quantity gives three numbers in three sources: field survey 22.42, billing summary 21.59, telemetry 17.01 m³; the model's true value is 21.31 m³ and no source sees it. - The subscriber count also changes with the source — 1,400, 886, 1,041 — and without writing which one was counted, this number cannot be read. - Within a source, the gap opened by a processing decision reaches 7.48 m³ and is larger than the gap between sources. - An estimate and a lower bound are written in the same form and cannot be told apart: 21.59 is an estimate, 14.11 is a lower bound, and both come out of the same summary. - Bias comes from the coverage decision: the survey has 55.9% of the fourth tier, the telemetry 58.8%, and the two sources are biased in opposite directions. #### Next Step This lesson measured which records the source leaves outside, but accepted without question the records let inside: all 544 survey rows were counted as filled in correctly, all 1,041 telemetry records as consistent. In a real set this does not hold. A reading may be physically impossible, a field may be left empty, two fields may contradict each other, a record may have been written months ago. The next lesson turns these four questions into four measurable dimensions and produces a number for each; then it tightens the thresholds and shows how many rows are eliminated and in which direction the eliminated rows push the reported number. ### Data Quality Dimensions Source: https://academia.sh/en/courses/introduction-to-data-analytics/data-quality-dimensions Each of the four quality dimensions gives a number: 46 records carry physically impossible readings, 200 records have an empty required field, 104 records have a tier field that contradicts the reading, 482 records are older than 90 days, and 730 records pass all four rules. As the threshold tightens, 1,400 rows fall to 422 and the reported mean drops from 24.70 to 20.83 m³, crossing the model's true value of 21.31 once on the way; the eliminated rows are not random, and 88.2% of the top tariff tier is eliminated. The previous lesson measured which records the source leaves outside, and accepted without question the records let inside. This lesson removes that assumption: the row that enters the record can itself be flawed. A reading may be physically impossible, a field may be left empty, two fields may contradict each other, a record may have been written months ago. The shared name of these four problems is **data quality**, and it cannot be reduced to a single number. Each one asks for a separate rule, a separate threshold, and a separate number. The measure of the lesson has two layers: a number for each dimension, and how many rows are eliminated when the threshold changes. The second layer is more important than the first, because a cleaned set is a smaller and **different** set. - **SD15.** The set is the 1,400-row subscriber record of the previous lessons, and this lesson adds flaws to it: corrupted readings, empty fields, and a stale tier field. The flaws are drawn **after** the base fields; the base set does not change. The set is **fictional**; the model's true value is 21.31 m³. - **SD16. Accuracy** is whether the reading gives the truth; here only physically impossible values can be caught. **Completeness** is the required fields being filled in. - **SD17. Consistency** is two fields agreeing with each other: the tariff tier stored separately in the record must be the same as the tier computed from the reading. **Timeliness** is the number of days that have passed since the last reading. - **SD18.** The accuracy threshold is 120 m³ a month for one meter; this value is given to the model as input. A floating-point reading is tested against a **threshold**, not for equality, because the measurement is truncated at two decimal places and the result of a comparison with an upper bound should not depend on equality. #### Four Dimensions, Four Numbers ```js // quality.mjs — the same set, this time four quality dimensions are measured. IT IS A MODEL: // the corruptions are drawn AFTER the base fields, the base set does not change. export const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40], UPPER_LIMIT = 120; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; export const REGION = [ { name: "north", share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; export const tierOf = (v) => (v < TARIFF[0] ? 1 : v < TARIFF[1] ? 2 : v < TARIFF[2] ? 3 : 4); export const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const truth = Math.round(t * 100) / 100; const meters = pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1; const visits = pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]); const satisfaction = pick(r(), b.sat) + 1; const temperature = Math.round((6 + r() * 22) * 10) / 10; const elapsed = Math.floor(Math.pow(r(), 1.9) * 210); // Corruptions. The chance of a missing field and a stale tier rises with consumption: // high-consumption meters sit at hard-to-reach points and change tiers more often. const fault = r() < 0.035, sign = r() < 0.4; const measured = fault ? (sign ? -truth : truth * 10) : truth; const regionEmpty = r() < 0.03; const satEmpty = r() < 0.04 + 0.0035 * truth; const stale = r() < 0.05 + 0.0022 * truth; return { subscriber_id: 10001 + i, region: regionEmpty ? null : b.name, consumption_true: truth, consumption_m3: measured, corrupted: fault, meter_count: meters, visit_count: visits, satisfaction: satEmpty ? null : satisfaction, temperature_c: temperature, days_elapsed: elapsed, tier_field: stale ? tierOf(truth * 0.8) : tierOf(truth) }; }); export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length; export const TRUTH = mean(RECORDS.map((k) => k.consumption_true)); // Four dimensions, four rules. The floating-point comparison is made with a threshold, not // equality: UPPER_LIMIT is the monthly upper bound for one meter, and a reading above it is // not physically possible. export const accurate = (k) => k.consumption_m3 > 0 && k.consumption_m3 < UPPER_LIMIT * k.meter_count; export const complete = (k) => k.region !== null && k.satisfaction !== null; export const consistent = (k) => k.tier_field === tierOf(k.consumption_m3); export const current = (k, days) => k.days_elapsed <= days; if (import.meta.url === `file://${process.argv[1]}`) { const y = (h) => String(h[0]).padEnd(14) + String(h[1]).padEnd(34) + h.slice(2).map((v) => String(v).padStart(10)).join(""); console.log(`model (seed ${SEED}): ${RECORD_COUNT} records; true value of the model: ` + `mean ${TRUTH.toFixed(2)} m3, raw mean ` + `${mean(RECORDS.map((k) => k.consumption_m3)).toFixed(2)} m3`); console.log("\n" + y(["dimension", "rule", "violations", "rate"])); const measures = [ ["accuracy", `reading from 0 to ${UPPER_LIMIT} x meters`, RECORDS.filter((k) => !accurate(k)).length], ["completeness", "region and satisfaction filled", RECORDS.filter((k) => !complete(k)).length], ["consistency", "tier field matches the reading", RECORDS.filter((k) => !consistent(k)).length], ["timeliness", "last reading newer than 90 days", RECORDS.filter((k) => !current(k, 90)).length], ]; for (const [name, rule, n] of measures) { console.log(y([name, rule, n, `${(100 * n / RECORD_COUNT).toFixed(1)}%`])); } const none = RECORDS.filter((k) => accurate(k) && complete(k) && consistent(k) && current(k, 90)).length; const corrupted = RECORDS.filter((k) => k.corrupted).length; console.log(`\ncorrupted readings in the model ${corrupted}, caught by the accuracy rule ` + `${RECORDS.filter((k) => !accurate(k)).length}`); console.log(`records passing all four rules ${none}, ` + `${(100 * none / RECORD_COUNT).toFixed(1)}%`); const y2 = (h) => String(h[0]).padEnd(16) + h.slice(1).map((v) => String(v).padStart(11)).join(""); console.log("\n" + y2(["field", "filled", "empty", "fill rate"])); for (const [name, f] of [["region", (k) => k.region], ["satisfaction", (k) => k.satisfaction], ["consumption_m3", (k) => k.consumption_m3], ["days_elapsed", (k) => k.days_elapsed]]) { const filled = RECORDS.filter((k) => f(k) !== null).length; console.log(y2([name, filled, RECORD_COUNT - filled, `${(100 * filled / RECORD_COUNT).toFixed(1)}%`])); } } ``` ``` model (seed 20260218): 1400 records; true value of the model: mean 21.31 m3, raw mean 24.70 m3 dimension rule violations rate accuracy reading from 0 to 120 x meters 46 3.3% completeness region and satisfaction filled 200 14.3% consistency tier field matches the reading 104 7.4% timeliness last reading newer than 90 days 482 34.4% corrupted readings in the model 58, caught by the accuracy rule 46 records passing all four rules 730, 52.1% field filled empty fill rate region 1357 43 96.9% satisfaction 1232 168 88.0% consumption_m3 1400 0 100.0% days_elapsed 1400 0 100.0% ``` The four dimensions gave four separate numbers and none of them replaces another. The largest violation is in timeliness — 482 records, 34.4% — and this number is not a data error: the records are correct, only old. The smallest violation is in accuracy, but its effect is the largest: the 46 corrupted readings push the raw mean from 21.31 up to 24.70 m³. **The number of violations and the effect of the violations do not move in the same direction**; a quality report does not end with counting violations, it also writes how far the violations moved the reported number. The note under the accuracy row shows the real limit. In the model 58 readings are corrupted and the rule catches 46 of them. The 12 records that escape belong to households with more than one meter: because the threshold is 120 m³ per meter, a corrupted reading of 210 m³ in a household with two meters stays under the limit and looks valid. **An accuracy rule catches only the impossible; a value that is wrong but possible cannot be told apart by any rule.** The only way to catch the 12 escaped records is to compare against a second source, and the previous lesson showed that source carries a bias of its own. The field fill table says completeness is measured per field. The `region` field is 96.9% filled, `satisfaction` is 88.0%; `consumption_m3` is never empty. Looked at per record, what remains is 730 rows that pass all four rules — only 52.1% of the set. The dimensions are measured separately but **applied together**, and the intersection is always smaller than even the smallest dimension. #### As the Threshold Tightens ```js // threshold.mjs — how many rows are eliminated as the quality threshold tightens, and which // way the reported number moves. Each dimension is applied alone first, then together. import { RECORDS, RECORD_COUNT, TRUTH, mean, accurate, complete, consistent, current, tierOf } from "./quality.mjs"; const y = (h) => String(h[0]).padEnd(28) + h.slice(1).map((v) => String(v).padStart(11)).join(""); const show = (name, l) => { const o = mean(l.map((k) => k.consumption_m3)); console.log(y([name, l.length, `${(100 * l.length / RECORD_COUNT).toFixed(1)}%`, o.toFixed(2), (o - TRUTH >= 0 ? "+" : "") + (o - TRUTH).toFixed(2)])); }; console.log(`true value of the model ${TRUTH.toFixed(2)} m3; every mean below is computed ` + `from the measured field`); console.log("\neach dimension alone"); console.log(y(["rule", "kept", "kept %", "mean", "deviation"])); show("no filter", RECORDS); show("accuracy only", RECORDS.filter(accurate)); show("completeness only", RECORDS.filter(complete)); show("consistency only", RECORDS.filter(consistent)); show("timeliness only (90 days)", RECORDS.filter((k) => current(k, 90))); const PROFILE = [ ["P0 no filter", () => true], ["P1 accuracy", (k) => accurate(k)], ["P2 + completeness, 90 days", (k) => accurate(k) && complete(k) && current(k, 90)], ["P3 + consistency, 60 days", (k) => accurate(k) && complete(k) && consistent(k) && current(k, 60)], ["P4 same rules, 30 days", (k) => accurate(k) && complete(k) && consistent(k) && current(k, 30)], ]; console.log("\nas the threshold tightens"); console.log(y(["profile", "kept", "kept %", "mean", "deviation"])); const kept = PROFILE.map(([name, f]) => { const l = RECORDS.filter(f); show(name, l); return l; }); console.log("\nelimination rate per tier (by the true reading)"); const y2 = (h) => String(h[0]).padEnd(28) + h.slice(1).map((v) => String(v).padStart(13)).join(""); const tier = (b) => RECORDS.filter((k) => tierOf(k.consumption_true) === b); console.log(y2(["profile", "eliminated", ...[1, 2, 3, 4].map((b) => `tier ${b} (${tier(b).length})`)])); PROFILE.forEach(([name, f]) => { const d = [1, 2, 3, 4].map((b) => `${(100 * tier(b).filter((k) => !f(k)).length / tier(b).length).toFixed(1)}%`); console.log(y2([name, RECORDS.filter((k) => !f(k)).length, ...d])); }); const p2 = kept[2], p4 = kept[4]; console.log(`\n${p2.length - p4.length} more rows were eliminated between P2 and P4, ` + `the mean moved ${(mean(p4.map((k) => k.consumption_m3)) - mean(p2.map((k) => k.consumption_m3))).toFixed(2)} m3`); console.log(`the cleanest set (P4) is ${p4.length} rows and deviates ` + `${(mean(p4.map((k) => k.consumption_m3)) - TRUTH).toFixed(2)} m3 from the true value of the model`); ``` ``` true value of the model 21.31 m3; every mean below is computed from the measured field each dimension alone rule kept kept % mean deviation no filter 1400 100.0% 24.70 +3.39 accuracy only 1354 96.7% 22.71 +1.41 completeness only 1200 85.7% 24.32 +3.01 consistency only 1296 92.6% 21.47 +0.17 timeliness only (90 days) 918 65.6% 24.01 +2.70 as the threshold tightens profile kept kept % mean deviation P0 no filter 1400 100.0% 24.70 +3.39 P1 accuracy 1354 96.7% 22.71 +1.41 P2 + completeness, 90 days 761 54.4% 22.37 +1.07 P3 + consistency, 60 days 608 43.4% 21.05 -0.25 P4 same rules, 30 days 422 30.1% 20.83 -0.48 elimination rate per tier (by the true reading) profile eliminated tier 1 (48) tier 2 (970) tier 3 (348) tier 4 (34) P0 no filter 0 0.0% 0.0% 0.0% 0.0% P1 accuracy 46 4.2% 3.5% 2.6% 2.9% P2 + completeness, 90 days 639 45.8% 44.4% 48.0% 55.9% P3 + consistency, 60 days 792 54.2% 55.4% 59.2% 67.6% P4 same rules, 30 days 978 66.7% 69.0% 71.0% 88.2% 339 more rows were eliminated between P2 and P4, the mean moved -1.55 m3 the cleanest set (P4) is 422 rows and deviates -0.48 m3 from the true value of the model ``` The first table measures the dimensions one by one and writes the difference in their effectiveness. The accuracy rule throws out 46 rows and brings the deviation from 3.39 down to 1.41 m³. The timeliness rule throws out 482 rows and brings the deviation from 3.39 down to only 2.70 m³. **There is no link between the number of rows thrown out and the accuracy gained**; the rule that throws out ten times more rows closes only about a fifth of the deviation. The consistency rule, on the other hand, throws out 104 rows and brings the deviation down to 0.17 m³, because most corrupted readings also contradict the tier field: two dimensions jointly catch part of the same rows, and the dimensions are not independent of each other. The second table tightens the threshold step by step, and the reported mean falls from 24.70 to 20.83 m³. On the way it **crosses** the model's true value of 21.31: 22.37 at P2, 21.05 at P3. The number closest to the truth is not in the cleanest set but somewhere in the middle, and only someone who knows the truth can see it. A real analysis has no such column; all it has is the "kept" and "mean" columns, and a person looking at them sees the number falling as the threshold tightens, not approaching the truth. The third table writes the reason for the fall. The elimination rate rises as the tier rises: in the P4 profile, 66.7% of the first tier is eliminated and 88.2% of the fourth. **The eliminated rows are not random.** Because in the model the chance of a missing field and a stale tier rises with consumption, every rule that raises quality eliminates the high consumers disproportionately and pulls the mean down. As the set gets cleaned it becomes **cleaner but more biased**. Two records follow from this, and a quality report should contain both. The first is how many rows were eliminated: from 1,400 to 422, 30.1%. The second is in which breakdown the eliminated rows concentrate; a one-line note saying "data cleaned" does not carry this information. A threshold is not a tuning knob but a decision that redefines the sample: the 20.83 m³ of P4 is not the mean of 1,400 subscribers but of the 422 subscribers whose records are recent and complete enough, and these two sentences are not the same thing. #### Summary - Data quality is measured with four separate dimensions, and each asks for its own rule, threshold, and number: 46 impossible readings, 200 incomplete records, 104 inconsistent tiers, 482 old records. - The number of violations and the effect of the violations do not move in the same direction: 46 corrupted readings pull the deviation from 3.39 to 1.41 m³, while 482 old records pull it only to 2.70. - An accuracy rule catches only the impossible: of the 58 corrupted readings in the model, 12 stay under the threshold and look valid. - As the threshold tightens the set falls from 1,400 rows to 422 and the mean drops from 24.70 to 20.83 m³, crossing the model's true value once. - The eliminated rows are not random: 88.2% of the top tariff tier is eliminated, so the cleaned set is cleaner and at the same time more biased. #### Next Step These four lessons asked what kind of thing the data is, which scale it stands on, where it comes from, and how reliable it is. Their shared assumption was this: the question is a question that can be asked, and the whole matter is finding the right answer. Each of the remaining 422 rows belongs to a household, and these records report many things, from consumption habits to the days the house stood empty. When a breakdown is thinned enough, the row is no longer a sample but a single person. The next lesson takes a set of questions and separates those that the data can **answer** from those that **should not be asked**; it counts how many people a breakdown falls down to and shows, with a measurable threshold, where the analysis should stop. ### Data Ethics and Privacy Source: https://academia.sh/en/courses/introduction-to-data-analytics/data-ethics-and-privacy Of 18 questions asked of the same subscriber set, 14 can be answered with the fields on hand and 4 have no field; when the boundary rule changes, the number of questions that are answerable but should not be asked rises from 3 to 8 and the reportable questions fall from 11 to 6. As the breakdown deepens the group shrinks: the smallest group is 209 records in the region breakdown and 1 record in the four-field breakdown, and 45 subscribers sit alone in a cell. When the minimum group size threshold is raised from 5 to 10, the suppressed cells rise from 117 to 146 and the records left outside the report from 245 to 431. The previous lesson tightened the quality threshold and left 422 rows out of 1,400. Each of those 422 rows belongs to a household. Throughout the course the rows were treated as observations; when a breakdown is thinned enough, the row is no longer an observation but a single person. This lesson counts two things. The first is how many of the questions that can be asked of the same set are **answerable but should not be asked**. The second is how many people a breakdown falls down to. What they share is this: neither is read from the data, because that a calculation can be done does not say that it should be done. In the Detection, Response and Compliance course, **what** an audit log records was measured. The question here is different: the record has already been taken; what is measured is **where the analysis will stop**. - **SD19.** The set is again the 1,400-row subscriber record and it is **fictional**. What is counted is not the content of the set but the questions that can be asked of it. - **SD20. Personal data**, as a kind, is a record that can be tied to a person. The boundaries are treated here as **analysis decisions**, and how far the decision moves the number is measured. - **SD21.** The three boundary rules are **cumulative**: no inference that descends to a single person is made, data is not used outside the purpose for which it was collected, and an attribute of a sensitive kind is not produced by inference. - **SD22. The minimum group size threshold** $k$ is the least number of records a breakdown cell must contain for it to be reported. $k$ is given to the model as input. #### Answerable but Not to Be Asked ```js // questions.mjs — 18 questions asked of the same subscriber set are classified on two axes: // can the data answer it, and does the boundary rule allow it to be asked. IT IS A MODEL. // [question, has field, reaches a person, outside purpose, sensitive inference] export const QUESTIONS = [ ["mean consumption per region", 1, 0, 0, 0], ["tariff tier distribution", 1, 0, 0, 0], ["share of subscribers with a field visit", 1, 0, 0, 0], ["satisfaction difference by region", 1, 0, 0, 0], ["mean monthly reading per meter", 1, 0, 0, 0], ["region with the highest reading delay", 1, 0, 0, 0], ["why the subscriber asked for a tariff change", 0, 0, 0, 0], ["how income level affects consumption", 0, 0, 0, 0], ["did the complaining subscriber leave the service", 0, 0, 0, 0], ["did those who saw the promotion consume less", 0, 0, 0, 0], ["which household is empty right now", 1, 1, 0, 0], ["a subscriber's daily living pattern", 1, 1, 0, 0], ["list of lowest-consumption subscribers", 1, 1, 0, 0], ["hand subscriber list to tariff promotion", 1, 0, 1, 0], ["route overdue subscribers to other services", 1, 0, 1, 0], ["evaluate the field team by satisfaction", 1, 0, 1, 0], ["household size and age from consumption", 1, 0, 0, 1], ["maintenance need from night consumption", 1, 0, 0, 1], ].map(([text, field, person, purpose, sensitive]) => ({ text, field, person, purpose, sensitive })); // The boundary rules are cumulative; each is added to the previous set. export const RULE = [ ["R0 no rule", () => false], ["R1 no identity descent", (q) => q.person === 1], ["R2 + purpose tie", (q) => q.person === 1 || q.purpose === 1], ["R3 + sensitive inference", (q) => q.person === 1 || q.purpose === 1 || q.sensitive === 1], ]; if (import.meta.url === `file://${process.argv[1]}`) { const N = QUESTIONS.length, answerable = QUESTIONS.filter((q) => q.field === 1).length; console.log(`${N} questions, same subscriber set; ${answerable} can be answered with the ` + `fields on hand, no field exists for ${N - answerable}`); const y = (h) => String(h[0]).padEnd(26) + h.slice(1).map((v) => String(v).padStart(15)).join(""); console.log("\n" + y(["boundary rule", "unaskable", "reportable", "closed share"])); for (const [name, barrier] of RULE) { const banned = QUESTIONS.filter((q) => q.field === 1 && barrier(q)).length; console.log(y([name, banned, answerable - banned, `${(100 * banned / answerable).toFixed(1)}%`])); } const [, r1] = RULE[1], [, r3] = RULE[3]; console.log(`\nunaskable at R1 ${QUESTIONS.filter((q) => q.field === 1 && r1(q)).length}, ` + `at R3 ${QUESTIONS.filter((q) => q.field === 1 && r3(q)).length}; ` + `the data did not change at all, only the boundary rule did`); console.log(`\ntwo separate boundaries: no field for ${N - answerable} questions (data boundary), ` + `${QUESTIONS.filter((q) => q.field === 1 && r3(q)).length} questions at R3 (analysis boundary)`); const y2 = (h) => String(h[0]).padEnd(46) + h.slice(1).map((v) => String(v).padStart(14)).join(""); console.log("\n" + y2(["questions hitting the analysis boundary", "answerable", "barrier"])); for (const q of QUESTIONS.filter((q) => q.field === 1 && r3(q))) { console.log(y2([q.text, "yes", q.person ? "identity" : q.purpose ? "purpose" : "sensitive"])); } } ``` ``` 18 questions, same subscriber set; 14 can be answered with the fields on hand, no field exists for 4 boundary rule unaskable reportable closed share R0 no rule 0 14 0.0% R1 no identity descent 3 11 21.4% R2 + purpose tie 6 8 42.9% R3 + sensitive inference 8 6 57.1% unaskable at R1 3, at R3 8; the data did not change at all, only the boundary rule did two separate boundaries: no field for 4 questions (data boundary), 8 questions at R3 (analysis boundary) questions hitting the analysis boundary answerable barrier which household is empty right now yes identity a subscriber's daily living pattern yes identity list of lowest-consumption subscribers yes identity hand subscriber list to tariff promotion yes purpose route overdue subscribers to other services yes purpose evaluate the field team by satisfaction yes purpose household size and age from consumption yes sensitive maintenance need from night consumption yes sensitive ``` The table puts two separate boundaries side by side. Four questions hit the **data boundary**: the set has no income, intervention, or churn record, and this boundary can be lifted by adding fields. Eight questions hit the **analysis boundary**, and the last column says "yes" for every one of them: they are answerable. This boundary is not lifted by adding fields, because the barrier is not missing data but the question itself. The number of questions that cannot be asked is not a constant either. With only the rule against descending to identity in force it is 3, with all three rules together it is 8; the reportable questions fall from 11 to 6, and not a single record in the set changes. An analysis cannot be satisfied with a yes or no to the question "does the data contain personal information"; unless it is written which rules were applied, it is not known how many questions were closed. The barriers of the eight closed questions differ from one another. Three descend to identity: the question "which household is empty right now" can be answered from a drop in consumption, and its answer points to a single household. Three break the tie to purpose: the records were collected for measurement and billing. Two produce an attribute of a sensitive kind **by inference**; the set has no age or maintenance-need field, but a guess can be produced from the night consumption pattern. This third group is the quietest, because there is no collected sensitive field — the sensitive attribute is born in the analysis itself. #### As the Breakdown Deepens, the Group Shrinks As long as the rule against descending to identity remains a statement of intent it cannot be measured; to be measurable it has to be tied to a number: how many people are in a breakdown cell. ```js // breakdown.mjs — group size as the breakdown deepens; the same 1,400-row // subscriber record. IT IS A MODEL, the set is fictional. const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40]; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; const REGION = [ { name: "north", share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const consumption = Math.round(t * 100) / 100; return { region: b.name, meter_count: pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1, visit_count: pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]), satisfaction: pick(r(), b.sat) + 1, tier: consumption < TARIFF[0] ? 1 : consumption < TARIFF[1] ? 2 : consumption < TARIFF[2] ? 3 : 4 }; }); const FIELDS = [ ["region", (k) => k.region, 5], ["tier", (k) => k.tier, 4], ["meter", (k) => k.meter_count, 4], ["satisfaction", (k) => k.satisfaction, 5], ["visit", (k) => k.visit_count, 6], ]; const cells = (d) => { const m = new Map(); for (const k of RECORDS) { const key = FIELDS.slice(0, d).map(([, f]) => f(k)).join("|"); m.set(key, (m.get(key) ?? 0) + 1); } return [...m.values()]; }; const y = (h) => String(h[0]).padEnd(48) + h.slice(1).map((v) => String(v).padStart(10)).join(""); console.log(`${RECORD_COUNT} subscriber records; the breakdown deepens by one field at each step`); console.log("\n" + y(["breakdown", "possible", "occupied", "smallest", "singleton", "k<5 rows"])); for (let d = 1; d <= FIELDS.length; d++) { const h = cells(d), possible = FIELDS.slice(0, d).reduce((a, [, , n]) => a * n, 1); console.log(y([FIELDS.slice(0, d).map(([a]) => a).join(" x "), possible, h.length, Math.min(...h), h.filter((n) => n === 1).length, h.filter((n) => n < 5).reduce((a, n) => a + n, 0)])); } // Threshold decision: minimum group size k. Cells below k are suppressed. const suppress = (d, k) => { const h = cells(d); const closed = h.filter((n) => n < k); return { cells: h.length, closed: closed.length, rows: closed.reduce((a, n) => a + n, 0) }; }; console.log("\nwhen the minimum group size threshold changes (breakdown: region x tier x meter x satisfaction)"); const y2 = (h) => String(h[0]).padEnd(10) + h.slice(1).map((v) => String(v).padStart(16)).join(""); console.log(y2(["threshold", "occupied cells", "suppressed", "rows outside", "coverage"])); for (const k of [5, 10, 20]) { const o = suppress(4, k); console.log(y2([`k = ${k}`, o.cells, o.closed, o.rows, `${(100 * (RECORD_COUNT - o.rows) / RECORD_COUNT).toFixed(1)}%`])); } const a = suppress(4, 5), b = suppress(4, 10); console.log(`\nwhen k goes from 5 to 10, suppressed cells rise from ${a.closed} to ${b.closed}, ` + `rows left outside the report from ${a.rows} to ${b.rows}`); const deep = cells(5); console.log(`in the finest breakdown ${deep.filter((n) => n === 1).length} subscribers sit alone in ` + `a cell; every number in that cell is the record of one person`); ``` ``` 1400 subscriber records; the breakdown deepens by one field at each step breakdown possible occupied smallest singleton k<5 rows region 5 5 209 0 0 region x tier 20 16 2 0 2 region x tier x meter 80 54 1 4 49 region x tier x meter x satisfaction 400 190 1 45 245 region x tier x meter x satisfaction x visit 2400 502 1 251 694 when the minimum group size threshold changes (breakdown: region x tier x meter x satisfaction) threshold occupied cells suppressed rows outside coverage k = 5 190 117 245 82.5% k = 10 190 146 431 69.2% k = 20 190 172 770 45.0% when k goes from 5 to 10, suppressed cells rise from 117 to 146, rows left outside the report from 245 to 431 in the finest breakdown 251 subscribers sit alone in a cell; every number in that cell is the record of one person ``` In the single-field breakdown the smallest group is 209 records and there is no singleton cell. With the second field the smallest group falls to 2, with the third the first singleton cells appear, with the fourth 45, and with the fifth 251 subscribers sit alone in a cell. **As the breakdown deepens, the number of cells grows by multiplication while the number of records stays fixed**; the ratio of the two is the group size, and it falls quickly to 1. What this means is the following: in the four-field breakdown, the line "mean consumption of subscribers in the north region, in the third tariff tier, with two meters, with a satisfaction score of 1" looks like a summary and is in fact the reading of a single person. The report carries no identity field; even so, for someone who knows those four attributes the line belongs to that person. **Descending to identity does not require an identity field.** The last table sets a measurable decision against this. When $k = 5$ is chosen, 117 of the 190 cells are suppressed, 245 records are left outside the report, and coverage falls to 82.5%; when $k = 10$ is chosen, 146 cells are suppressed, 431 records are left outside, and coverage is 69.2%. Doubling the threshold raises the suppressed cells by 25% but **raises the records left outside by 76%**, because as the threshold rises, more crowded cells also fall under the limit. $k$ is not a technical setting but a decision that writes down a trade-off, and two defensible thresholds produce two different reports. If a breakdown table does not state next to it with which $k$ it was suppressed, the reader assumes the suppressed cells do not exist. #### Summary - There are two separate boundaries: four questions hit the data boundary and can be opened by adding fields, and eight questions hit the analysis boundary and are all answerable. - The number of questions that cannot be asked also depends on a decision: it is 3 with one rule and 8 with three rules together. - A sensitive attribute is not always a collected field; in two questions it is not in the set and is produced by inference. - As the breakdown deepens the group shrinks: the smallest group is 209 records in one field and 1 record in four fields, and 45 subscribers sit alone in a cell. - The minimum group size threshold is a trade-off decision: when $k$ goes from 5 to 10, the suppressed cells rise from 117 to 146 and the records left outside from 245 to 431. #### Course Wrap-Up Nine lessons did the same job: produced a number, wrote the decision, changed the decision and printed the difference. | Lesson | Reported number | Changed decision | Difference | |---|---|---|---| | What Is Data Analytics | 3,849 active subscribers | the definition of active changes | 3,548, divergence on 645 records | | Types of Analytics | 7 answerable questions | the derived field is ignored | 2 questions, 5 of them close | | Analysis Workflow | 33 step visits | the decision is written at the step where it arises, a preview is added | 8 visits, 1 rework loop | | Analyst Roles | -5.14% subscribers | the criterion is fee per subscriber | +1.89% | | Data Types | mean 1.20 visits | the discrete field is read as a distribution | records equal to it 0, 68.3% below | | Measurement Scales | east ranks first at 3.71 | the satisfaction step coding changes | east ranks fifth at 5.88 | | Data Sources | 22.42 m³ field survey | the source switches to telemetry | 17.01 m³, 5.41 m³ gap | | Data Quality Dimensions | 24.70 m³, 1,400 rows | the quality threshold tightens | 20.83 m³, 422 rows | | Data Ethics and Privacy | 3 questions cannot be asked | the boundary rule widens | 8 questions, 6 of them reportable | None of the nine rows has a calculation error. Both the left column and the right column of every row are defensible numbers, and the only thing that produces the difference between them is the middle column. This is the rule of the course, and it has been measured nine times: **if a number does not state next to it the decision with which it was produced, that number is not a finding but a choice.** A number whose decision is not written is counted as unmeasured. The table itself shows a problem. All the decisions counted in the nine lessons were taken **by hand**: which definition was chosen, which coding was used, which source was read, which threshold was applied. None of them stands in a written place; the body of the decisions is in the text of these lessons, not in the calculation itself. The question of whether a second person taking the same set will find the same number has no answer, because what that person has is not the list of decisions but only the result. The next course, **Analysis with Spreadsheets**, takes the work from here: in a setup where the calculation stands in one place and changing an input recalculates all the results, it measures how much of the same decisions stays visible.