--- title: Sampling source: 'https://academia.sh/en/courses/data-collection-and-preparation/sampling' course: 'Data Collection and Preparation' language: en updated: '2026-10-07T04:31:11+00:00' license: 'CC BY-SA 4.0' objectives: - 'Draws four samples from the same population with four sampling decisions and compares the answer each gives to the same question with the answer of the population' - 'Measures the representation error of a sample on several dimensions in percentage points and counts who the records left outside are' - 'Shows how sample size narrows the estimate range by repeating the same decision with different seeds' --- # Sampling Four samples drawn from a population of 1,877 rows give four different answers to the same two questions: the convenience sample gives a mean of 19.99 m³ and a top-tier share of 33.0%, the simple random sample 20.04 m³ and 32.0%, the sample stratified by period 20.50 m³ and 35.0%, and the sample whose draw unit is the subscriber 20.70 m³ and 35.0%; the population's own answer is 20.61 m³ and 35.4%. The region representation error of the convenience sample is 0.0 points, and that is exactly why it inspires confidence — the period representation error of the same sample is 25.4 points, because all 477 rows of the second period and 1,200 of the 1,400 subscribers were left outside. The same random decision gives, over twelve seeds, a range 3.90 m³ wide for n=50 and 0.71 m³ wide for n=800: the error of a single sample cannot be known, its spread is measured. In four lessons the data came by four routes, and how many rows were lost on each route was counted. The common assumption of these lessons was that the whole set could be collected: the file could be read, the query could run, the interface could be paged to the end, the page could be parsed from start to finish. When the population grows, this assumption falls, and a decision takes its place — a subset is taken instead of all the records. The measure of sampling is not whether the subset is correct; a subset is not the population and always gives a different number. What is measured is **representation**: how accurately the sample reflects which dimension of the population, and who the rows left outside are. All four decisions of this lesson are defensible, and all four give a different answer to the same question. - **DC20.** The population is fiction and is the same as the set of the previous four lessons: 1,877 reading rows, 1,400 subscribers, 38,690.7 m³. The mean per reading row is 20.61 m³, that is, the same number computed in the second lesson of this course. **It is a model.** - **DC21.** The generator is this lesson's own generator and its seed is visible (20240). With the same seed the output is the same on every run; the mixing does not change anything in place, it works on a copy of the population. - **DC22.** The tariff tiers split at 10 and 25 m³, and the comparison is made on the deci-m³ integer; this way the boundary values never enter floating-point equality. - **DC23.** **Representation error** is the largest absolute difference between the sample share and the population share on the same dimension, and it is written in percentage points. - **DC24.** The threshold for counting two means as the same is 0.05 m³, because readings are kept with one decimal and a difference below that can only come from rounding. ## The Population's Own Answer ```js // population.mjs — the population the sampling will work on. IT IS A MODEL: the same 1877 // reading rows as in the previous four lessons, generated with the same seed. The // tariff tiers split at 10 and 25 m3; the comparison is made on the deci-m3 integer, // so the boundary values never enter floating-point equality. export const SEED = 4711; export const rng = (t) => { let s = t >>> 0; return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; }; export const REGION = ["north", "east", "central", "south", "west"]; export const POPULATION = []; const r = rng(SEED); for (let i = 0; i < 1877; i++) { // first the first period of 1400 subscribers, then 477 second const n = i < 1400 ? i : i - 1400, deci = 50 + Math.floor(r() * 321); POPULATION.push({ subscriber_no: 10000 + n, period: i < 1400 ? "2024-01" : "2024-02", deci, region: REGION[n % 5], tier: deci <= 100 ? "under 10" : deci <= 250 ? "10-25" : "over 25" }); } export const m3 = (k) => k.deci / 10; export const mean = (set) => set.reduce((t, k) => t + m3(k), 0) / set.length; export const topShare = (set) => 100 * set.filter((k) => k.tier === "over 25").length / set.length; // The category shares of a dimension (percent). export const share = (set, field) => { const s = new Map(); for (const k of set) s.set(k[field], (s.get(k[field]) ?? 0) + 1); return new Map([...s].map(([a, n]) => [a, 100 * n / set.length])); }; // Representation error: the largest absolute difference between the sample share and // the population share on the same dimension, in percentage points. export const error = (set, field) => { const o = share(set, field), k = share(POPULATION, field); return Math.max(...[...k.keys()].map((a) => Math.abs((o.get(a) ?? 0) - k.get(a)))); }; if (import.meta.url === `file://${process.argv[1]}`) { console.log(`seed ${SEED}; population ${POPULATION.length} reading rows, ` + `${new Set(POPULATION.map((k) => k.subscriber_no)).size} subscribers, ` + `${POPULATION.reduce((t, k) => t + m3(k), 0).toFixed(1)} m3`); console.log(`answer of the population: mean ${mean(POPULATION).toFixed(2)} m3, ` + `share of the top tier ${topShare(POPULATION).toFixed(1)}%`); for (const field of ["period", "region", "tier"]) console.log(`${field.padEnd(8)}` + [...share(POPULATION, field)] .map(([a, p]) => `${a} ${p.toFixed(1)}`).join(" | ")); } ``` ``` seed 4711; population 1877 reading rows, 1400 subscribers, 38690.7 m3 answer of the population: mean 20.61 m3, share of the top tier 35.4% period 2024-01 74.6 | 2024-02 25.4 region north 20.0 | east 20.0 | central 20.0 | south 20.0 | west 20.0 tier 10-25 46.6 | under 10 18.0 | over 25 35.4 ``` This lesson differs from the others because the answer of the population is known: the sampling error is not estimated, it is measured directly. In a real study this line does not exist, and since the right answer is not known, the only thing that can be checked is **the decision itself**. The lower three lines are the structure of the population on three dimensions, and they are the scale the sample will be compared against. The period dimension is unbalanced: 74.6% of the rows come from the first period, because only 477 of the 1,400 subscribers have a second period reading. The region dimension, on the other hand, is almost perfectly balanced, the share of the five regions is around 20%. This distinction determines the rest of the lesson: a sample can look correct on a balanced dimension without any effort. ## Four Sampling Decisions ```js // sample.mjs — four sampling decisions from the same population. The generator is this // lesson's own generator and the seed is visible; with the same seed the output is the // same on every run. What is measured is not whether the sample is "correct" but how // well it represents the population on which dimension and who the rows left outside are. import { POPULATION, rng, m3, mean, topShare, error } from "./population.mjs"; const N = 200, SAMPLE_SEED = 20240; const mix = (array, seed) => { const g = rng(seed), a = [...array]; for (let i = a.length - 1; i > 0; i--) { const j = Math.floor(g() * (i + 1)); [a[i], a[j]] = [a[j], a[i]]; } return a; }; // O1: the first N rows of the list. O2: simple random sample. O3: proportional // sample stratified by period — from each stratum as many rows are taken as its share in the population. const O1 = (n) => POPULATION.slice(0, n); const O2 = (n, seed = SAMPLE_SEED) => mix(POPULATION, seed).slice(0, n); const O3 = (n, seed = SAMPLE_SEED) => ["2024-01", "2024-02"].flatMap((d) => { const t = POPULATION.filter((k) => k.period === d); return mix(t, seed + d.length).slice(0, Math.round(n * t.length / POPULATION.length)); }); // O4: the draw unit is the subscriber, not the row. N subscribers are chosen and // all rows of the chosen subscribers are taken; the number of rows that come out is // determined not by the decision but by the number of rows per subscriber. const O4 = (n, seed = SAMPLE_SEED) => { const inside = new Set(mix([...new Set(POPULATION.map((k) => k.subscriber_no))], seed).slice(0, n)); return POPULATION.filter((k) => inside.has(k.subscriber_no)); }; const THRESHOLD = 0.05; // the threshold for counting two means as the same const y = (h) => h.map((v, j) => (j ? String(v).padStart([0, 9, 10, 8, 9, 8, 8, 8, 8][j]) : String(v).padEnd(28))).join(""); console.log(`population: mean ${mean(POPULATION).toFixed(2)} m3, top tier ` + `${topShare(POPULATION).toFixed(1)}%; last three columns representation error (percentage points)`); console.log(y(["sampling decision", "rows in", "rows out", "mean", "diff m3", "top %", "period", "region", "tier"])); const result = {}; for (const [name, f] of [["O1 convenience (first 200)", O1], ["O2 simple random", O2], ["O3 stratified (period)", O3], ["O4 subscriber unit", O4]]) { const o = result[name] = f(N); console.log(y([name, POPULATION.length, o.length, mean(o).toFixed(2), (mean(o) - mean(POPULATION)).toFixed(2), topShare(o).toFixed(1), error(o, "period").toFixed(1), error(o, "region").toFixed(1), error(o, "tier").toFixed(1)])); } // The same decision, twelve seeds: the error of a single sample cannot be known, its spread is measured. const y2 = (h) => h.map((v, j) => (j ? String(v).padStart([0, 8, 10, 10, 9, 14][j]) : String(v).padEnd(12))).join(""); console.log(`\nO2, twelve seeds (${SAMPLE_SEED}-${SAMPLE_SEED + 11})`); console.log(y2(["sample", "seeds", "minimum", "maximum", "range", "top % range"])); for (const n of [50, 200, 800]) { const t = [], u = []; for (let i = 0; i < 12; i++) { const o = O2(n, SAMPLE_SEED + i); t.push(mean(o)); u.push(topShare(o)); } console.log(y2([`n = ${n}`, 12, Math.min(...t).toFixed(2), Math.max(...t).toFixed(2), (Math.max(...t) - Math.min(...t)).toFixed(2), (Math.max(...u) - Math.min(...u)).toFixed(1)])); } const o1 = result["O1 convenience (first 200)"], o3 = result["O3 stratified (period)"]; const insideSubscribers = new Set(o1.map((k) => k.subscriber_no)); console.log(`\n${POPULATION.length - o1.length} rows remained outside O1: all ` + `${POPULATION.filter((k) => k.period === "2024-02").length} rows of the second period and ` + `${new Set(POPULATION.map((k) => k.subscriber_no)).size - insideSubscribers.size} of the ` + `${new Set(POPULATION.map((k) => k.subscriber_no)).size} subscribers in the population`); console.log(`The mean of O1 differs from the population by ${Math.abs(mean(o1) - mean(POPULATION)).toFixed(2)} ` + `m3, ` + `${Math.round(Math.abs(mean(o1) - mean(POPULATION)) / THRESHOLD)} times the ${THRESHOLD} m3 threshold; but the region representation ` + `error is ${error(o1, "region").toFixed(1)} points, that is, the sample looks flawless on one dimension`); console.log(`O3 brings the period dimension down to ${error(o3, "period").toFixed(1)} points, but its error on the region ` + `dimension is ${error(o3, "region").toFixed(1)} points: stratification guarantees only the ` + `dimension it stratifies on`); const o4 = result["O4 subscriber unit"]; console.log(`O4 selected ${N} subscribers but produced ${o4.length} rows: ${o4.length - N} of the selected ` + `subscribers also have a second period reading. Because the draw unit is the subscriber, the mean per ` + `subscriber, ${(o4.reduce((s, k) => s + m3(k), 0) / N).toFixed(2)} m3, can be read directly; the mean per ` + `subscriber of the population is ` + `${(POPULATION.reduce((s, k) => s + m3(k), 0) / new Set(POPULATION.map((k) => k.subscriber_no)).size) .toFixed(2)} m3`); ``` ``` population: mean 20.61 m3, top tier 35.4%; last three columns representation error (percentage points) sampling decision rows in rows out mean diff m3 top % period region tier O1 convenience (first 200) 1877 200 19.99 -0.63 33.0 25.4 0.0 3.5 O2 simple random 1877 200 20.04 -0.58 32.0 1.1 2.5 3.4 O3 stratified (period) 1877 200 20.50 -0.11 35.0 0.1 2.0 0.5 O4 subscriber unit 1877 274 20.70 0.09 35.0 1.6 3.7 0.5 O2, twelve seeds (20240-20251) sample seeds minimum maximum range top % range n = 50 12 18.41 22.32 3.90 22.0 n = 200 12 19.26 21.55 2.29 11.5 n = 800 12 20.33 21.04 0.71 4.3 1677 rows remained outside O1: all 477 rows of the second period and 1200 of the 1400 subscribers in the population The mean of O1 differs from the population by 0.63 m3, 13 times the 0.05 m3 threshold; but the region representation error is 0.0 points, that is, the sample looks flawless on one dimension O3 brings the period dimension down to 0.1 points, but its error on the region dimension is 2.0 points: stratification guarantees only the dimension it stratifies on O4 selected 200 subscribers but produced 274 rows: 74 of the selected subscribers also have a second period reading. Because the draw unit is the subscriber, the mean per subscriber, 28.36 m3, can be read directly; the mean per subscriber of the population is 27.64 m3 ``` All four decisions were drawn from the same 1,877 rows, and all four gave a different answer to the same two questions. The mean varies between 19.99 and 20.70 m³, the share of the top tier between 32.0% and 35.0%. No decision made a calculation error; all four gave the correct mean of their own set. O1 is the lesson's trap. The region representation error is **0.0 points** — the five regions are in the sample at exactly the same ratio as their share in the population. Because the list is ordered by subscriber number and the region derives from the subscriber number, the first 200 rows come out flawless on this dimension. The period representation error of the same sample, however, is **25.4 points**: the whole sample is from the first period, and **all 477 rows** of the second period and **1,200** of the 1,400 subscribers were left outside. A sample looking flawless on one dimension does not show that it is correct on other dimensions, and balanced dimensions feed this fallacy. The mean errors of O2 and O1 are close to each other: 0.58 and 0.63 m³, both more than ten times the threshold. The difference between them is not the size of the error but its **auditability**. The lower table gives the error of O2 as a spread: the same decision over twelve seeds produces values between 18.41 and 22.32 m³ for n=50, between 19.26 and 21.55 for n=200, and between 20.33 and 21.04 for n=800. The range goes down from 3.90 to 0.71 m³, and the range in the share of the top tier falls from 22.0 points to 4.3 points. The error of a random decision cannot be known in a single sample, but its spread can be measured and narrowed by enlarging the sample. The error of O1, on the other hand, does not depend on the seed; it depends on the order of the list, it cannot be narrowed by trying again, and it cannot be expressed as a range. O3 corrects the dimension where the population is unbalanced. The period representation error falls from 25.4 points to **0.1 points**, the mean error to 0.11 m³, and the share of the top tier to 35.0% — the closest value to the population's 35.4%. But the region representation error of the same sample is **2.0 points**, that is, larger than that of O1. Stratification is not a seal of quality, it is a targeted guarantee: it secures only the dimension it stratifies on and leaves the remaining dimensions to chance. Which dimension to stratify on depends on in which breakdown the question the sample will answer is going to be read. O4 changes the draw unit itself and produces the only irregular row in the table: 200 subscribers were selected but the set came out as **274 rows**, because 74 of the selected subscribers also have a second period reading. The number of rows was determined not by the decision but by the number of rows per subscriber. In return, this sample gives the mean per subscriber directly — 28.36 m³, against 27.64 m³ for the population. In the second lesson of this course the unit of the row was a query problem; here it is a sampling problem, and the answer is the same: **the draw unit must be the same as the row of the report.** A sample drawn at the row level gives a subscriber with two readings two chances of being selected and silently weights a question per subscriber. ## Summary - The measure of sampling is not the correctness of the subset but representation: the largest difference between the sample share and the population share, written dimension by dimension. - Four samples drawn from the same population give the mean as 19.99, 20.04, 20.50, and 20.70 m³; the population's answer is 20.61 m³, and none of the four decisions made a calculation error. - The convenience sample gives 0.0 points of error on the region dimension, and that is exactly why it inspires confidence; the period error of the same sample is 25.4 points, because all 477 rows of the second period and 1,200 of the 1,400 subscribers were left outside. - The error of a random decision is not known in a single sample, its spread is measured: over twelve seeds the range is 3.90 m³ for n=50 and 0.71 m³ for n=800. The error of the convenience sample cannot be narrowed this way. - Stratification guarantees only the dimension it stratifies on: while the period error falls to 0.1 points, the region error stays at 2.0 points. - The draw unit must be the same as the row of the report: a sample of 200 subscribers produces 274 rows and gives the mean per subscriber directly, while a sample drawn at the row level silently weights the same question. ## Next Step Over five lessons a single account was kept. The file gave 2,076 physical rows and one parsing decision silently dropped 142 rows; the query returned 1,877 rows and a join dropped 766 rows and multiplied 669 rows; the interface, with a narrow scope, never gave 477 rows, and pagination skipped 35 records and brought 35 records twice; when its structure changed, the publication page dropped all 1,877 rows in one version and made them be read from the wrong field in another. The 1,677 rows of this lesson, on the other hand, are not a loss but **a written decision**, and the difference between the two is the rule of the course: a cleaning whose loss you cannot write down is not a cleaning. But all five lessons counted the rows **from the outside**. How many rows came, how many rows dropped, how many rows came twice — all of it was read over the number of records. The **inside** of the rows that came was never opened: whether a field was empty, whether the same subscriber appeared twice with two different spellings, whether a date was corrupted while being converted from text to number was never looked at. The next topic goes inside the row and keeps the same account, this time at the field level: when missing values are deleted, which group is eliminated disproportionately, which records a similarity threshold merges wrongly, how many records a silent coercion corrupts.