--- title: 'Data Ethics and Privacy' source: 'https://academia.sh/en/courses/introduction-to-data-analytics/data-ethics-and-privacy' course: 'Introduction to Data Analytics' language: en updated: '2026-10-07T04:31:12+00:00' license: 'CC BY-SA 4.0' objectives: - 'Tells two separate boundaries apart: a question the data cannot answer and a question that is answerable but should not be asked' - 'Counts by running how the number of unaskable questions moves when the boundary rule changes' - 'Measures how group size falls as the breakdown deepens and how many records land in a single cell' - 'Compares the cells suppressed and the records left outside the report at two minimum group size thresholds' --- # Data Ethics and Privacy Of 18 questions asked of the same subscriber set, 14 can be answered with the fields on hand and 4 have no field; when the boundary rule changes, the number of questions that are answerable but should not be asked rises from 3 to 8 and the reportable questions fall from 11 to 6. As the breakdown deepens the group shrinks: the smallest group is 209 records in the region breakdown and 1 record in the four-field breakdown, and 45 subscribers sit alone in a cell. When the minimum group size threshold is raised from 5 to 10, the suppressed cells rise from 117 to 146 and the records left outside the report from 245 to 431. The previous lesson tightened the quality threshold and left 422 rows out of 1,400. Each of those 422 rows belongs to a household. Throughout the course the rows were treated as observations; when a breakdown is thinned enough, the row is no longer an observation but a single person. This lesson counts two things. The first is how many of the questions that can be asked of the same set are **answerable but should not be asked**. The second is how many people a breakdown falls down to. What they share is this: neither is read from the data, because that a calculation can be done does not say that it should be done. In the Detection, Response and Compliance course, **what** an audit log records was measured. The question here is different: the record has already been taken; what is measured is **where the analysis will stop**. - **SD19.** The set is again the 1,400-row subscriber record and it is **fictional**. What is counted is not the content of the set but the questions that can be asked of it. - **SD20. Personal data**, as a kind, is a record that can be tied to a person. The boundaries are treated here as **analysis decisions**, and how far the decision moves the number is measured. - **SD21.** The three boundary rules are **cumulative**: no inference that descends to a single person is made, data is not used outside the purpose for which it was collected, and an attribute of a sensitive kind is not produced by inference. - **SD22. The minimum group size threshold** $k$ is the least number of records a breakdown cell must contain for it to be reported. $k$ is given to the model as input. ## Answerable but Not to Be Asked ```js // questions.mjs — 18 questions asked of the same subscriber set are classified on two axes: // can the data answer it, and does the boundary rule allow it to be asked. IT IS A MODEL. // [question, has field, reaches a person, outside purpose, sensitive inference] export const QUESTIONS = [ ["mean consumption per region", 1, 0, 0, 0], ["tariff tier distribution", 1, 0, 0, 0], ["share of subscribers with a field visit", 1, 0, 0, 0], ["satisfaction difference by region", 1, 0, 0, 0], ["mean monthly reading per meter", 1, 0, 0, 0], ["region with the highest reading delay", 1, 0, 0, 0], ["why the subscriber asked for a tariff change", 0, 0, 0, 0], ["how income level affects consumption", 0, 0, 0, 0], ["did the complaining subscriber leave the service", 0, 0, 0, 0], ["did those who saw the promotion consume less", 0, 0, 0, 0], ["which household is empty right now", 1, 1, 0, 0], ["a subscriber's daily living pattern", 1, 1, 0, 0], ["list of lowest-consumption subscribers", 1, 1, 0, 0], ["hand subscriber list to tariff promotion", 1, 0, 1, 0], ["route overdue subscribers to other services", 1, 0, 1, 0], ["evaluate the field team by satisfaction", 1, 0, 1, 0], ["household size and age from consumption", 1, 0, 0, 1], ["maintenance need from night consumption", 1, 0, 0, 1], ].map(([text, field, person, purpose, sensitive]) => ({ text, field, person, purpose, sensitive })); // The boundary rules are cumulative; each is added to the previous set. export const RULE = [ ["R0 no rule", () => false], ["R1 no identity descent", (q) => q.person === 1], ["R2 + purpose tie", (q) => q.person === 1 || q.purpose === 1], ["R3 + sensitive inference", (q) => q.person === 1 || q.purpose === 1 || q.sensitive === 1], ]; if (import.meta.url === `file://${process.argv[1]}`) { const N = QUESTIONS.length, answerable = QUESTIONS.filter((q) => q.field === 1).length; console.log(`${N} questions, same subscriber set; ${answerable} can be answered with the ` + `fields on hand, no field exists for ${N - answerable}`); const y = (h) => String(h[0]).padEnd(26) + h.slice(1).map((v) => String(v).padStart(15)).join(""); console.log("\n" + y(["boundary rule", "unaskable", "reportable", "closed share"])); for (const [name, barrier] of RULE) { const banned = QUESTIONS.filter((q) => q.field === 1 && barrier(q)).length; console.log(y([name, banned, answerable - banned, `${(100 * banned / answerable).toFixed(1)}%`])); } const [, r1] = RULE[1], [, r3] = RULE[3]; console.log(`\nunaskable at R1 ${QUESTIONS.filter((q) => q.field === 1 && r1(q)).length}, ` + `at R3 ${QUESTIONS.filter((q) => q.field === 1 && r3(q)).length}; ` + `the data did not change at all, only the boundary rule did`); console.log(`\ntwo separate boundaries: no field for ${N - answerable} questions (data boundary), ` + `${QUESTIONS.filter((q) => q.field === 1 && r3(q)).length} questions at R3 (analysis boundary)`); const y2 = (h) => String(h[0]).padEnd(46) + h.slice(1).map((v) => String(v).padStart(14)).join(""); console.log("\n" + y2(["questions hitting the analysis boundary", "answerable", "barrier"])); for (const q of QUESTIONS.filter((q) => q.field === 1 && r3(q))) { console.log(y2([q.text, "yes", q.person ? "identity" : q.purpose ? "purpose" : "sensitive"])); } } ``` ``` 18 questions, same subscriber set; 14 can be answered with the fields on hand, no field exists for 4 boundary rule unaskable reportable closed share R0 no rule 0 14 0.0% R1 no identity descent 3 11 21.4% R2 + purpose tie 6 8 42.9% R3 + sensitive inference 8 6 57.1% unaskable at R1 3, at R3 8; the data did not change at all, only the boundary rule did two separate boundaries: no field for 4 questions (data boundary), 8 questions at R3 (analysis boundary) questions hitting the analysis boundary answerable barrier which household is empty right now yes identity a subscriber's daily living pattern yes identity list of lowest-consumption subscribers yes identity hand subscriber list to tariff promotion yes purpose route overdue subscribers to other services yes purpose evaluate the field team by satisfaction yes purpose household size and age from consumption yes sensitive maintenance need from night consumption yes sensitive ``` The table puts two separate boundaries side by side. Four questions hit the **data boundary**: the set has no income, intervention, or churn record, and this boundary can be lifted by adding fields. Eight questions hit the **analysis boundary**, and the last column says "yes" for every one of them: they are answerable. This boundary is not lifted by adding fields, because the barrier is not missing data but the question itself. The number of questions that cannot be asked is not a constant either. With only the rule against descending to identity in force it is 3, with all three rules together it is 8; the reportable questions fall from 11 to 6, and not a single record in the set changes. An analysis cannot be satisfied with a yes or no to the question "does the data contain personal information"; unless it is written which rules were applied, it is not known how many questions were closed. The barriers of the eight closed questions differ from one another. Three descend to identity: the question "which household is empty right now" can be answered from a drop in consumption, and its answer points to a single household. Three break the tie to purpose: the records were collected for measurement and billing. Two produce an attribute of a sensitive kind **by inference**; the set has no age or maintenance-need field, but a guess can be produced from the night consumption pattern. This third group is the quietest, because there is no collected sensitive field — the sensitive attribute is born in the analysis itself. ## As the Breakdown Deepens, the Group Shrinks As long as the rule against descending to identity remains a statement of intent it cannot be measured; to be measurable it has to be tied to a number: how many people are in a breakdown cell. ```js // breakdown.mjs — group size as the breakdown deepens; the same 1,400-row // subscriber record. IT IS A MODEL, the set is fictional. const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40]; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; const REGION = [ { name: "north", share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] }, { name: "south", share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] }, { name: "east", share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] }, { name: "west", share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] }, { name: "central", share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] }, ]; const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const consumption = Math.round(t * 100) / 100; return { region: b.name, meter_count: pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1, visit_count: pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]), satisfaction: pick(r(), b.sat) + 1, tier: consumption < TARIFF[0] ? 1 : consumption < TARIFF[1] ? 2 : consumption < TARIFF[2] ? 3 : 4 }; }); const FIELDS = [ ["region", (k) => k.region, 5], ["tier", (k) => k.tier, 4], ["meter", (k) => k.meter_count, 4], ["satisfaction", (k) => k.satisfaction, 5], ["visit", (k) => k.visit_count, 6], ]; const cells = (d) => { const m = new Map(); for (const k of RECORDS) { const key = FIELDS.slice(0, d).map(([, f]) => f(k)).join("|"); m.set(key, (m.get(key) ?? 0) + 1); } return [...m.values()]; }; const y = (h) => String(h[0]).padEnd(48) + h.slice(1).map((v) => String(v).padStart(10)).join(""); console.log(`${RECORD_COUNT} subscriber records; the breakdown deepens by one field at each step`); console.log("\n" + y(["breakdown", "possible", "occupied", "smallest", "singleton", "k<5 rows"])); for (let d = 1; d <= FIELDS.length; d++) { const h = cells(d), possible = FIELDS.slice(0, d).reduce((a, [, , n]) => a * n, 1); console.log(y([FIELDS.slice(0, d).map(([a]) => a).join(" x "), possible, h.length, Math.min(...h), h.filter((n) => n === 1).length, h.filter((n) => n < 5).reduce((a, n) => a + n, 0)])); } // Threshold decision: minimum group size k. Cells below k are suppressed. const suppress = (d, k) => { const h = cells(d); const closed = h.filter((n) => n < k); return { cells: h.length, closed: closed.length, rows: closed.reduce((a, n) => a + n, 0) }; }; console.log("\nwhen the minimum group size threshold changes (breakdown: region x tier x meter x satisfaction)"); const y2 = (h) => String(h[0]).padEnd(10) + h.slice(1).map((v) => String(v).padStart(16)).join(""); console.log(y2(["threshold", "occupied cells", "suppressed", "rows outside", "coverage"])); for (const k of [5, 10, 20]) { const o = suppress(4, k); console.log(y2([`k = ${k}`, o.cells, o.closed, o.rows, `${(100 * (RECORD_COUNT - o.rows) / RECORD_COUNT).toFixed(1)}%`])); } const a = suppress(4, 5), b = suppress(4, 10); console.log(`\nwhen k goes from 5 to 10, suppressed cells rise from ${a.closed} to ${b.closed}, ` + `rows left outside the report from ${a.rows} to ${b.rows}`); const deep = cells(5); console.log(`in the finest breakdown ${deep.filter((n) => n === 1).length} subscribers sit alone in ` + `a cell; every number in that cell is the record of one person`); ``` ``` 1400 subscriber records; the breakdown deepens by one field at each step breakdown possible occupied smallest singleton k<5 rows region 5 5 209 0 0 region x tier 20 16 2 0 2 region x tier x meter 80 54 1 4 49 region x tier x meter x satisfaction 400 190 1 45 245 region x tier x meter x satisfaction x visit 2400 502 1 251 694 when the minimum group size threshold changes (breakdown: region x tier x meter x satisfaction) threshold occupied cells suppressed rows outside coverage k = 5 190 117 245 82.5% k = 10 190 146 431 69.2% k = 20 190 172 770 45.0% when k goes from 5 to 10, suppressed cells rise from 117 to 146, rows left outside the report from 245 to 431 in the finest breakdown 251 subscribers sit alone in a cell; every number in that cell is the record of one person ``` In the single-field breakdown the smallest group is 209 records and there is no singleton cell. With the second field the smallest group falls to 2, with the third the first singleton cells appear, with the fourth 45, and with the fifth 251 subscribers sit alone in a cell. **As the breakdown deepens, the number of cells grows by multiplication while the number of records stays fixed**; the ratio of the two is the group size, and it falls quickly to 1. What this means is the following: in the four-field breakdown, the line "mean consumption of subscribers in the north region, in the third tariff tier, with two meters, with a satisfaction score of 1" looks like a summary and is in fact the reading of a single person. The report carries no identity field; even so, for someone who knows those four attributes the line belongs to that person. **Descending to identity does not require an identity field.** The last table sets a measurable decision against this. When $k = 5$ is chosen, 117 of the 190 cells are suppressed, 245 records are left outside the report, and coverage falls to 82.5%; when $k = 10$ is chosen, 146 cells are suppressed, 431 records are left outside, and coverage is 69.2%. Doubling the threshold raises the suppressed cells by 25% but **raises the records left outside by 76%**, because as the threshold rises, more crowded cells also fall under the limit. $k$ is not a technical setting but a decision that writes down a trade-off, and two defensible thresholds produce two different reports. If a breakdown table does not state next to it with which $k$ it was suppressed, the reader assumes the suppressed cells do not exist. ## Summary - There are two separate boundaries: four questions hit the data boundary and can be opened by adding fields, and eight questions hit the analysis boundary and are all answerable. - The number of questions that cannot be asked also depends on a decision: it is 3 with one rule and 8 with three rules together. - A sensitive attribute is not always a collected field; in two questions it is not in the set and is produced by inference. - As the breakdown deepens the group shrinks: the smallest group is 209 records in one field and 1 record in four fields, and 45 subscribers sit alone in a cell. - The minimum group size threshold is a trade-off decision: when $k$ goes from 5 to 10, the suppressed cells rise from 117 to 146 and the records left outside from 245 to 431. ## Course Wrap-Up Nine lessons did the same job: produced a number, wrote the decision, changed the decision and printed the difference. | Lesson | Reported number | Changed decision | Difference | |---|---|---|---| | What Is Data Analytics | 3,849 active subscribers | the definition of active changes | 3,548, divergence on 645 records | | Types of Analytics | 7 answerable questions | the derived field is ignored | 2 questions, 5 of them close | | Analysis Workflow | 33 step visits | the decision is written at the step where it arises, a preview is added | 8 visits, 1 rework loop | | Analyst Roles | -5.14% subscribers | the criterion is fee per subscriber | +1.89% | | Data Types | mean 1.20 visits | the discrete field is read as a distribution | records equal to it 0, 68.3% below | | Measurement Scales | east ranks first at 3.71 | the satisfaction step coding changes | east ranks fifth at 5.88 | | Data Sources | 22.42 m³ field survey | the source switches to telemetry | 17.01 m³, 5.41 m³ gap | | Data Quality Dimensions | 24.70 m³, 1,400 rows | the quality threshold tightens | 20.83 m³, 422 rows | | Data Ethics and Privacy | 3 questions cannot be asked | the boundary rule widens | 8 questions, 6 of them reportable | None of the nine rows has a calculation error. Both the left column and the right column of every row are defensible numbers, and the only thing that produces the difference between them is the middle column. This is the rule of the course, and it has been measured nine times: **if a number does not state next to it the decision with which it was produced, that number is not a finding but a choice.** A number whose decision is not written is counted as unmeasured. The table itself shows a problem. All the decisions counted in the nine lessons were taken **by hand**: which definition was chosen, which coding was used, which source was read, which threshold was applied. None of them stands in a written place; the body of the decisions is in the text of these lessons, not in the calculation itself. The question of whether a second person taking the same set will find the same number has no answer, because what that person has is not the list of decisions but only the result. The next course, **Analysis with Spreadsheets**, takes the work from here: in a setup where the calculation stands in one place and changing an input recalculates all the results, it measures how much of the same decisions stays visible.