Unit 03 of 03
Ethics and Responsibility
Counting who is harmed by a system whose measure looks good: measuring data, label and deployment bias separately and seeing all three hit the same number by different paths, the impossibility of satisfying demographic parity, equal opportunity and predictive parity at the same time and how much fixing one breaks the others, how much adding or removing a single record moves the output, and reading side by side what is lost from accuracy and what is lost from recall as the noise level is swept, why anonymization alone is not enough, how many of the questions of a person who inherits a model are answered by the documentation, the number moved by the single item that is not pinned and the total number of items accumulated across the curriculum.
Published Updated
Lessons
- 01 Sources of Bias Counting separately that the same harm arises in three different places: the linear model measured on thirteen subgroups gives 0.8194 accuracy and 0.3704 recall overall, while recall stays at 0.2222 for subscribers with few readings. Because of the 69 subscribers who never enter the table, the share of few-reading subscribers reads 0.3349 instead of 0.3695; a training set built only from full-reading subscribers lowers precision in that subgroup from 0.5455 to 0.3774, the 19 suspicious cases that never reach the record drop recall to 0.0370, and a capacity of forty visits applied to the same subgroup at a rate of 0.0359 leaves recall at 0.1111. In all three, overall accuracy moves by at most 0.0337. 6 min read
- 02 Fairness Metrics Counting that three fairness criteria cannot be satisfied at the same time: with a base rate of 0.4578 in the east region against 0.1663 in the other four regions, a single threshold gives a parity gap of 0.3984, an opportunity gap of 0.4842 and a predictive gap of 0.1242. While the same number of subscribers is flagged, tuning the threshold pair to demographic parity raises the predictive gap to 0.4800, tuning it to equal opportunity raises it to 0.4262, and tuning it to predictive parity raises the opportunity gap to 0.7038. The identity that the selection rate is the product of recall and base rate divided by precision is verified, and when two criteria are equalized the ratio of selection rates is shown to lock to the ratio of base rates, 2.7535; with capacity fixed there is no threshold pair that gives all three together, and the best attempt leaves a gap of 0.2727. 11 min read
- 03 Privacy-Preserving Learning Counting from the defense side what the model says about its own training records: in a 1,260-row table with the subscriber ID deleted, four subgroup axes produce 72 combinations and 6 records stand alone, and when the sixth field is added this number rises to 1,242, that is, to 0.9857. Removing a single training record moves the output of the linear model by at most 0.0313 and changes two predictions; the same record moves the output by 0.7778 and changes 14 predictions in the model built on the cell table, because 13 of the 66 cells hold a single record. When noise is swept over ten levels, masking the trace of the linear model asks for 0.04 noise and 0.0022 of accuracy, while that of the cell model asks for 0.80 noise and falling 0.1669 below the baseline. 12 min read
- 04 Model Documentation Counting which of the questions of the person who takes over the model are closed by a written statement: the same model gives 0.7778 accuracy overall on the test set and 0.0198 above the baseline, while in 12 measurable subgroups of the 14 built on four axes the best reads 0.9688 and the worst 0.6111, the subgroup gap is 0.3576 and three subgroups fall below their own baseline. Of the handover question set of twenty-two questions, the statement that carries only the number measured overall closes 5, the statement that adds the subgroup table closes 12, and the statement that also writes the usage limit closes 18; the remaining 4 questions close in no version because none of them is a measurement question. In five usage decisions the first version produces 4 wrong decisions and the second 1, while the third produces none because it can say unknown for the unmeasured subgroup. 6 min read
- 05 Reproducibility Counting the items that must be fixed for the same number to be produced a second time, and how much a single unfixed item moves it: 6 of the evaluation's own 9 items each move the test count on their own, the largest shift being 0.0370 or 7 subscribers; 8 of the same items move the subgroup gap, and the largest shift is 0.0989, 2.7 times what is read overall. When the minimum subgroup size is released, neither headline number changes at all but the measured subgroups fall from 12 to 11. On top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course, the 11 of the Visualization, Reporting and Analysis at Scale course, the 4 of the Data Preparation and Feature Engineering course and the 8 of the Deep Learning course, the evaluation's 9 items are added and the full list of the curriculum reaches 80. The course wrap-up gathers the fifteen lessons overall and in the worst subgroup, and the curriculum wrap-up gathers the eight courses and 121 lessons of the Machine Learning and Data Science curriculum in a single table. 12 min read