Unit 02 of 02
Quality and Principles
Turning quality dimensions into running checks and counting both what each check catches and its own flaw: the detection lag of six corruption classes and the three classes a flawed check set never catches; a partition check that stays green even though one period of one region was never written, because it looks at the region set and not at the period; the rows dropped by a schema change, the stage at which a data contract catches it, and the cost of the contract itself; a pipeline that is not idempotent duplicating rows on reprocessing, and the served number staying unchanged because the deduplication stage absorbs the duplicates; the stage and row count one must visit to go from a corrupted served value back to the source, compared with and without a lineage record; how many of the five stages carry an owner, and who receives a corruption caught at an ownerless stage; and the number of periods a retention policy makes impossible to audit retroactively.
Published Updated
Lessons
- 01 Data Quality Dimensions Turning quality dimensions into running checks and counting both what each check catches and its own flaw: the detection lag of the six corruption classes is 4, 0, 0, 4, 0 and 0 stages in order; with a flawed check set three classes are never caught; because the partition check looks at the region set, it stays green when a single period of one region was never written; a unit change moves ten cells of the summary and leaves the served values exactly the same. 5 min read
- 02 Schema Contracts Counting whom the schema contract between producer and consumer breaks and what the contract itself costs: when a field is renamed, the producer side has 0 faulty rows while on the consumer side 200 rows are dropped at screening, 200 more at consumption, and the summary goes from 55 cells to 45; once the contract is written the same corruption is caught at stage 0 but the output is not repaired; a strict contract that pins the field set exactly blocks 1200 good rows with no corruption present and drops the summary to 25 cells; batch-level enforcement blocks 2200 good rows and yields no output at all, while an adapter stage returns the serving to the baseline and grows the pipeline to six stages. 14 min read
- 03 Resilience to Repetition and Reprocessing Measuring that processing the same load a second time duplicates rows and that the served number does not show it: in an appending store, backfilling one period takes the raw rows from 2400 to 2600 and a full rerun takes them to 4800, reaching 7200 in three runs, yet the served value stays at 1212.55 in all three runs because the deduplication stage absorbs the surplus; when the resent load carries a correction, the same stage throws the correction away and leaves the value at 794.36; and when deduplication looks at the whole row instead of the key, the same reruns carry the value to 2425.09 and 3637.64. 14 min read
- 04 Data Lineage Counting the rows one has to look at to go from a corrupted served value back to the source, with and without a lineage record: a five-stage pipeline carries 9,460 rows across six layers while the lineage record holds 11,455 links, because every consumption row has two ancestors; without the record, explaining a served value means reading 9,260 rows and that number does not depend on which value one starts from, while the same search with the record ends between 94 and 155 rows. Two runs give two different loss stages: for late-arriving records the loss is at the source, for a schema change it is at the screening stage. The size of the serving deviation does not give the size of the corruption: when five times as many rows are corrupted, the serving moves less and three values go up. 10 min read
- 05 Data Governance Counting the ownership gap and the retention policy: in a pipeline with six positions counting the source, three of the five stages carry an owner, yet all six corruptions are caught at an ownerless position, because the checks run only at the source and at the summary layer. A finding walks downstream until it reaches an owner, and three of the six findings go to an owner who can repair them; in the other three the recipient is one to five stages away from the repair point and has no access to the evidence. The three classes whose repair point is ownerless are exactly the three classes the flawed check set never catches. A six-period retention window deletes six periods but raises the periods that can no longer be audited retroactively to seven, the summary falls from 55 cells to 25, 2200 of the lineage's 4400 consumption links point to a deleted row, and the correct partition check raises a false alarm on intact data. The lesson is the course wrap-up. 15 min read