Unit 01 of 03
Data Collection
Counting how many rows the data loses on its way into the analysis environment: reading the same file with two parsing decisions and the rows that silently drop because of quoted fields, line breaks, and encoding; a query's returned row not matching the unit of analysis and a join multiplying rows; the record skipped by consuming pagination wrongly and the collection time set by a queue that obeys the rate limit; the number of selectors that break when the page structure changes and the limit of access permission; the three answers that three sampling decisions give to the same question.
Published Updated
Lessons
- 01 File Formats The export file that produces the previous course's 2,037-row page is read with two parsing decisions, and two different sets come out: the quote-preserving read extracts all 2,037 records, while the naive split extracts 1,934 records from 2,076 physical lines and drops 142 lines without any warning — 78 half-lines split at a line break inside the address, 64 lines whose fields shifted. Of the remaining 1,934 records, 33 survive but carry a house number shifted out of the address into the reading field; the totals diverge by 984.2 m³, and the region name of 1,157 records does not match the list because the file was decoded with the wrong encoding. With the same 2,037 records held in a columnar structure, learning the total reading means reading 4,074 bytes instead of 108,230. 15 min read
- 02 Pulling Data from Databases Three query forms against the same store return three different row sets, and none of them raises an error: joining reading with subscriber gives 1,877 rows; when the visit table is added with an inner join, 766 rows silently drop and 669 rows multiply — about 76% of the input is corrupted, yet the net row loss is only 97 and the total deviates by just 7.0%, so someone looking at the numbers cannot see the damage. The left join drops no rows but raises the set to 2,546 rows and inflates the total 1.34-fold. The same question gives three numbers under three decisions: 20.61 m³ per reading row, 27.64 m³ per subscriber, 20.21 m³ in the set spread over visit rows. 13 min read
- 03 Collecting Data from Interfaces Three collection decisions against the same interface produce three different sets, and all three responses come back ok: a narrow-scope token collects 1,400 records and leaves out the 477 rows of the past period without ever requesting them, offset-based pagination collects exactly 1,877 rows but never receives 35 records and receives 35 records twice, and cursor-based pagination collects all 1,877 records once. The overall total of the offset-based decision matches exactly at 38,690.7 m³ because the skipped and duplicated records carry the same reading value; the damage shows up only in the region breakdown, as a 70.5 m³ deviation. Under a rate limit, the queue that counts the limit response as an empty page stops at 600 records and loses 1,277 records; the queue that obeys the limit collects the same set completely with 38 requests and 3 wait units, and when the page size is raised to 200 the wait drops to zero. 15 min read
- 04 Structural Parsing A page that publishes the same 1,877 rows is read in four structure versions with two selector sets, and the number of broken selectors turns out not to measure the damage: when a single wrapper layer is inserted, the path selector of the positional set breaks and all 1,877 rows drop, but the failure is loud. When a column is inserted, no selector breaks, 1,877 records come out, none drops, and every one is read from the wrong field; the total deviates by only 60.1 m³, 0.16%, and the region field of 1,877 records is no longer even a region name. When the presentation layer is renewed, this time two selectors of the semantic set break and the records out drop to zero. Comparing the total the page itself reports with the computed total is the only check that catches the silent shift. 14 min read
- 05 Sampling Four samples drawn from a population of 1,877 rows give four different answers to the same two questions: the convenience sample gives a mean of 19.99 m³ and a top-tier share of 33.0%, the simple random sample 20.04 m³ and 32.0%, the sample stratified by period 20.50 m³ and 35.0%, and the sample whose draw unit is the subscriber 20.70 m³ and 35.0%; the population's own answer is 20.61 m³ and 35.4%. The region representation error of the convenience sample is 0.0 points, and that is exactly why it inspires confidence — the period representation error of the same sample is 25.4 points, because all 477 rows of the second period and 1,200 of the 1,400 subscribers were left outside. The same random decision gives, over twelve seeds, a range 3.90 m³ wide for n=50 and 0.71 m³ wide for n=800: the error of a single sample cannot be known, its spread is measured. 14 min read