Unit 02 of 02
The Structure of Data
Counting what kind of thing the data being worked on is and which summaries that makes legitimate: the meaningless average produced by treating a discrete field as continuous, the summaries that nominal, ordinal, interval, and ratio scales permit, a primary and a secondary source giving different numbers for the same quantity, the number of rows eliminated by accuracy, completeness, consistency, and timeliness thresholds, the question that should not be asked even though the data can answer it, and the breakdown that descends to a single person.
Published Updated
Lessons
- 01 Data Types Eight fields of 1,400 subscriber records are stored as numbers and all eight get an average, but three of them are qualitative: the average of the region code comes out 3.19 under the register coding and 2.69 under the size coding, and not a single record in the data has changed. No single record equals the 1.20 average of the discrete visit field, and 68.3% of the records fall below that average; when the continuous consumption field is rounded to 5 m³ steps the total shifts by only 1.4 per thousand, yet 229 records change tariff tier. 9 min read
- 02 Measurement Scales Ten fields are placed on four scales and all 40 cells formed by multiplying them with four summaries are computed: 28 turn out to be legitimate, and 12 are numbers that were computed but that the scale does not support. Two defensible codings given to the ordinal satisfaction field — 1, 2, 3, 4, 5 and 0, 4, 6, 7, 8 — reverse the ranking of the regions: the first region falls to fifth without a single grade changing. On an interval scale the ratio of the same two readings falls from 4.67 to 1.08 when the zero point changes, while their difference stays fixed at 22.0 units. 13 min read
- 03 Data Sources The same quantity — average monthly consumption — gives three different numbers from three sources: 22.42 m³ from the field survey, 21.59 m³ for the value rebuilt from the billing summary, 17.01 m³ from meter telemetry, while the model's true value is 21.31 m³. Even the subscriber count comes out 1,400, 886, and 1,041 depending on the source. When a single decision is changed in each source, the six numbers spread between 14.11 and 22.42 m³ and none of them contains a calculation error; the source of the bias is visible in the coverage table. 12 min read
- 04 Data Quality Dimensions Each of the four quality dimensions gives a number: 46 records carry physically impossible readings, 200 records have an empty required field, 104 records have a tier field that contradicts the reading, 482 records are older than 90 days, and 730 records pass all four rules. As the threshold tightens, 1,400 rows fall to 422 and the reported mean drops from 24.70 to 20.83 m³, crossing the model's true value of 21.31 once on the way; the eliminated rows are not random, and 88.2% of the top tariff tier is eliminated. 13 min read
- 05 Data Ethics and Privacy Of 18 questions asked of the same subscriber set, 14 can be answered with the fields on hand and 4 have no field; when the boundary rule changes, the number of questions that are answerable but should not be asked rises from 3 to 8 and the reportable questions fall from 11 to 6. As the breakdown deepens the group shrinks: the smallest group is 209 records in the region breakdown and 1 record in the four-field breakdown, and 45 subscribers sit alone in a cell. When the minimum group size threshold is raised from 5 to 10, the suppressed cells rise from 117 to 146 and the records left outside the report from 245 to 431. 8 min read