--- title: 'Data Engineering and Data Science Distinction' source: 'https://academia.sh/en/courses/introduction-to-data-engineering/data-engineering-and-data-science-distinction' course: 'Introduction to Data Engineering' language: en updated: '2026-10-21T05:03:00+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the boundary between data engineering and data science as the stage of the pipeline from which the measure is read.' - 'Measures the same six corruption classes separately with the row trace and the model performance measures and shows that the two measures rank them differently.' - 'Shows with numbers that a corruption can improve the model measure and justifies why a measure not getting worse is not proof of health.' - 'Explains with the telescoping sum identity why the unit change is invisible in both measures and compares that it becomes visible when read on the period axis.' --- # Data Engineering and Data Science Distinction Printing the same six corruption classes side by side on the measure of two roles: the data engineer sees the row trace of five stages, while the model builder sees the mean absolute deviation (baseline 10.832 m3) of a forecast rule that takes the per-region mean served by the pipeline as its feature. The two measures rank the same six classes differently: schema_changed is the noisiest class in the engineer's trace (2200 rows fall to 1800) yet moves the model's deviation by only 0.003; late_arrival removes only 40 rows from the source yet moves the deviation by 0.105, shifting it the most; missing_partition lowers the deviation by 0.003, that is, the corruption improves the measure. unit_changed and type_drift leave no trace in either. The invisibility of the unit change comes from a telescoping identity: the sum of the eleven cells of a region equals the difference of the end indexes, and the corruption does not touch the ends; when the same table is read on the period axis, the July mean comes out as 40592074.02 instead of 26.7. The previous two lessons read the pipeline from a single eye. In production there is no single eye around this pipeline. There is a role that writes the stages, looks at the trace table and is alerted when a row drops; there is another role that takes the table at the end of the pipeline as a feature table, builds a rule or a model on it and looks at the performance number. The two stand at **different places** of the same pipeline. The role distinction itself is not the subject of this curriculum; the boundaries of responsibility were drawn separately in the Introduction to Data Analytics, Introduction to Machine Learning and Introduction to AI Engineering courses. The only thing this lesson adds is **seating the distinction on the stages of the pipeline**: from which stage the number of which role is read, and who measures the stage that remains between the two numbers. When this question is asked, an unpleasant result appears: the same corruption reflects differently on the two measures, the two rank the six classes differently, and one class leaves no trace in **either of them**. - **DL16.** The pipeline, the source, the six corruption classes and the seed are the same as in the previous two lessons. - **DL17. The procedure for role definitions is not repeated**; it was measured in the three introductory courses above. Here the distinction is written only as **the stage from which the measure is read**. - **DL18.** The measure of the data engineer is the **row trace**: the remaining counts of the five stages. This is the same as the table in the first lesson. - **DL19.** The measure of the model builder is the **performance**: the **mean absolute deviation**, in `m3`, of a forecast rule that takes the per-region mean served by the pipeline as its feature. The rule divides the served value of a region by the subscriber count of that region and gives this as the forecast to every subscriber in the region. - **DL20.** The test set is the periods `2026-11` and `2026-12` (**400 rows**). These two periods are not corrupted in any of the six classes; so every difference in the deviation comes from the model's **feature side**, not from the test side. - **DL21.** The model builder takes the feature table **from the pipeline** and does not reach the source. This is the boundary in production: the served layer is the contract between the two roles. - **DL22. The procedure for feature engineering and model evaluation is not repeated**; it was measured in the Data Preparation and Feature Engineering and Model Evaluation, Interpretability and Ethics courses. The model is deliberately at its simplest here. The aim is not for the model to be good, but **what its measure is sensitive to**. - **DL23.** No check; the baseline is the uninstrumented pipeline, the corruption caught is **0**, the lag is **undefined**. The pipeline is deterministic, a second run gives the same table. Time is not written. ## Two Measures, the Same Pipeline The block below builds the pipeline and the six corruption classes as they are, adds the model builder's measure on top, and prints two numbers side by side for each class: is the row trace the same as the intact pipeline, is the mean absolute deviation the same as the intact pipeline. ```python # FICTIONAL pipeline, FICTIONAL corruption classes and a FICTIONAL forecast rule. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value # region values stay Turkish: the summary table below is printed in sorted order # dogu/bati/kuzey/guney/merkez = east/west/north/south/central REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records CORRUPTIONS = ["late_arrival", "duplicated", "schema_changed", "unit_changed", "missing_partition", "type_drift"] def corrupt(records, cls, period="2026-07"): """Corrupts a single period , the remaining periods stay intact.""" out, counter = [], 0 for k in records: if k["period"] != period: out.append(k) continue if cls == "late_arrival": if k["subscriber"] <= "A040": counter += 1 continue out.append(k) elif cls == "duplicated": out.append(k) if k["subscriber"] <= "A060": out.append(dict(k)) counter += 1 elif cls == "schema_changed": y = dict(k) y["index_value"] = y.pop("index") counter += 1 out.append(y) elif cls == "unit_changed": y = dict(k) y["index"] = k["index"] * 1000 # liters instead of m3 y["unit"] = "m3" # LABEL DID NOT CHANGE counter += 1 out.append(y) elif cls == "missing_partition": if k["region"] == "kuzey": counter += 1 continue out.append(k) elif cls == "type_drift": y = dict(k) y["index"] = str(k["index"]) counter += 1 out.append(y) return out, counter def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped def s4_summary(records): totals = {} for k in records: key = (k["region"], k["period"]) totals[key] = totals.get(key, 0) + k["m3"] return totals, 0 def s5_serve(summary): region = {} for (b, d), v in summary.items(): region.setdefault(b, []).append(v) return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0 STAGES = [("screen", s1_screen), ("dedupe", s2_dedupe), ("consumption", s3_consumption), ("summary", s4_summary), ("serve", s5_serve)] def pipeline(records): trace, data = [], records for name, f in STAGES: data, dropped = f(data) n = len(data) if hasattr(data, "__len__") else 0 trace.append({"stage": name, "remaining": n, "dropped": dropped}) return data, trace TEST_PERIODS = ["2026-11", "2026-12"] def model_metric(records): """Model builder , takes the per-region mean served by the pipeline as a feature.""" v1, _ = s1_screen(records) v2, _ = s2_dedupe(v1) t, _ = s3_consumption(v2) summ, _ = s4_summary(t) features, _ = s5_serve(summ) n = {} for k in t: n.setdefault(k["region"], set()).add(k["subscriber"]) forecast = {b: features[b] / len(n[b]) for b in features} test = [k for k in t if k["period"] in TEST_PERIODS] return round(sum(abs(k["m3"] - forecast[k["region"]]) for k in test) / len(test), 3), len(test) source = generate_source() base_trace = [a["remaining"] for a in pipeline(source)[1]] base_dev, base_n = model_metric(source) print("test rows", base_n, " baseline deviation", base_dev, " m3") print(f"{'label':<19}{'row trace':>26}{'rows':>7}{'deviation':>11}{'diff':>8}{'model':>7}") print(f"{'intact':<19}{str(base_trace):>26}{'-':>7}{base_dev:>11}{'-':>8}{'-':>7}") for cls in CORRUPTIONS: trace = [a["remaining"] for a in pipeline(corrupt(source, cls)[0])[1]] dev, _ = model_metric(corrupt(source, cls)[0]) print(f"{cls:<19}{str(trace):>26}{('yes' if trace == base_trace else 'no'):>7}" + f"{dev:>11}{round(dev - base_dev, 3):>8}" + f"{('yes' if dev == base_dev else 'no'):>7}") ``` ``` test rows 400 baseline deviation 10.832 m3 label row trace rows deviation diff model intact [2400, 2400, 2200, 55, 5] - 10.832 - - late_arrival [2360, 2360, 2120, 55, 5] no 10.937 0.105 no duplicated [2460, 2400, 2200, 55, 5] no 10.832 0.0 yes schema_changed [2200, 2200, 1800, 45, 5] no 10.835 0.003 no unit_changed [2400, 2400, 2200, 55, 5] yes 10.832 0.0 yes missing_partition [2354, 2354, 2108, 53, 5] no 10.829 -0.003 no type_drift [2400, 2400, 2200, 55, 5] yes 10.832 0.0 yes ``` ## One Who Sees Rows and One Who Sees Performance The two measures rank the same six classes differently, and this difference in ranking is the real number of the role distinction. In the data engineer's trace the noisiest class is `schema_changed`: the consumption output falls from 2200 to **1800**, the summary from 55 to **45**. The change of a field name takes away a sixth of the source, and this is a drop that cannot be overlooked on a dashboard table. The same class moves the model builder's measure by **0.003**, that is, a relative difference of 0.0003 over 10.832. When the model builder looks at this number, they think nothing happened, and they are right: even if a sixth of the training data is lost, the per-region mean comes out almost the same from the remaining nine periods. The opposite direction is more interesting. `late_arrival` removes only **40 rows** from the source, that is, 0.0167 of 2400 (the smallest step of the row ratio is 0.0004, so this ratio can be read to three digits); this is a loss that falls below the band of most row count checks. In return it moves the model builder's deviation by **0.105**, and among the six classes it is the one that shifts it the most. The reason is not in the table but in the definition of the pipeline: the missing rows are not randomly distributed, they are concentrated in the first forty subscribers and pulled the region means down together. A loss that is **small in number but one-directional** is more harmful than a loss that is large in number but balanced, and the row count does not write this distinction. The third row is the cleanest distinction between the two roles. `duplicated` is plainly visible in the engineer's trace: the `screen` output is **2460** instead of 2400. In the model builder's measure, however, the difference is **exactly zero**, because the deduplication stage drops the sixty double records and the intact set reaches the feature table. This is a case where the pipeline does its job right; still, the engineer needs to see it, because the next double record from the same producer may not be caught at the same place. The `missing_partition` row shows the most dangerous side of reading a measure. The deviation **falls** from 10.832 to **10.829**, that is, the corruption **improves** the model's measure. The improvement is not real; since two periods of region `kuzey` (north) are completely lost, the served mean of that region is computed from the remaining nine cells and, by chance, falls slightly closer to the test periods. The result is this: **a measure not getting worse is not proof of health, because the corruption can improve the measure too.** The direction of the sign says nothing about the presence of corruption. ## What Is Visible in Neither The last reading of the table is two rows: `unit_changed` and `type_drift`. In both, the row trace is **exactly the same** as the intact pipeline, and the deviation is **exactly the same**. The routine measure of both roles is completely blind to these two classes. The reason is superficial for `type_drift`: the pipeline turns the text into a number, the conversion succeeds, and neither the rows nor the values change. The only thing that sees it is a type check that **never looks at the value** of the row, and such a check can run at the zeroth step of the pipeline. The reason is structural for `unit_changed`, and the block below shows it. ```python def region_total(records): v1, _ = s1_screen(records) v2, _ = s2_dedupe(v1) t, _ = s3_consumption(v2) summ, _ = s4_summary(t) total = {} for (b, d), v in summ.items(): total[b] = total.get(b, 0) + v return total def period_mean(records): v1, _ = s1_screen(records) v2, _ = s2_dedupe(v1) t, _ = s3_consumption(v2) d = {} for k in t: d.setdefault(k["period"], []).append(k["m3"]) return {a: round(sum(v) / len(v), 2) for a, v in d.items()} first = {k["subscriber"]: k["index"] for k in source if k["period"] == PERIODS[0]} last = {k["subscriber"]: k["index"] for k in source if k["period"] == PERIODS[-1]} s_top, b_top = region_total(source), region_total(corrupt(source, "unit_changed")[0]) print("region ends diff intact summary corrupted summary") for b in sorted(s_top): ends = sum(last[a] - first[a] for a, v in SUBSCRIBERS.items() if v["region"] == b) print(f"{b:<9}{ends:>11}{s_top[b]:>16}{b_top[b]:>19}") s_d, b_d = period_mean(source), period_mean(corrupt(source, "unit_changed")[0]) print("period intact corrupted") for d in PERIODS[5:9]: print(f"{d:<10}{s_d[d]:>10}{b_d[d]:>15}") ``` ``` region ends diff intact summary corrupted summary bati 9164 9164 9164 dogu 9695 9695 9695 guney 11721 11721 11721 kuzey 13338 13338 13338 merkez 14536 14536 14536 period intact corrupted 2026-06 24.08 24.08 2026-07 26.7 40592074.02 2026-08 25.55 -40592021.77 2026-09 28.96 28.96 ``` The three columns of the first table give **the same number** for every region. The left column was computed without going through the pipeline at all, directly from the end points of the source: the last period index of the region's subscribers minus the first period index. The middle and right columns are the sum of the eleven summary cells of the pipeline, with the intact and the corrupted source. The equality of the three is not a measurement result but an **identity**: the sum of successive differences equals the difference of the ends. Because `unit_changed` does not touch the ends, the total cannot change, and the served value, which divides the total by eleven, cannot change either. This is a result known even without a run; the run has only verified it. The second table reads the same feature table **on the period axis**, and there the corruption shouts: the July mean is **40592074.02** instead of 26.7, the August mean is **negative 40592021.77** instead of 25.55. The data is not hiding it; what hides it is **both roles summing over the period axis**. The engineer sums the rows per stage, the model builder sums the values per region; the corruption lives exactly on the axis under these two sums, with two signs that cancel each other. The rule that follows from here is general and applicable: **a measure that sums the axis on which the corruption lives cannot see that corruption.** The count of the role distinction can also be read from here. The engineer's measure is read from the first, second and third stages; the model builder's measure from the output of the fifth stage. Between them stands the **fourth stage**, and the routine measure of neither role reads those fifty-five cells. The place where the corruption sits is exactly there. The boundary is not a question of job definition; it is a question of **which number is read from which stage of the pipeline**, and as long as a stage remains unmeasured between the measures of the two roles, the boundary produces a gap. ## Summary - The boundary between data engineering and data science is not a job definition but **the stage from which the measure is read**: the engineer reads the row trace of the first to third stages, the model builder reads the output of the fifth stage. - The two measures rank the same six classes differently: `schema_changed` is the noisiest class in the trace (2200 rows fall to 1800) yet moves the deviation by only 0.003; `late_arrival` removes only 40 rows yet moves the deviation by 0.105, shifting it the most. - A loss that is small in number but one-directional is more harmful than a loss that is large in number but balanced, and the row count does not write this distinction. - `missing_partition` **lowers** the deviation from 10.832 to 10.829: a corruption can improve the measure, so a measure not getting worse is not proof of health. - `unit_changed` and `type_drift` give exactly the same result in both measures. The invisibility of the unit change comes from a telescoping identity: the sum of the eleven cells of a region equals the difference of the end indexes, and the corruption does not touch the ends. - When the same table is read on the period axis, the July mean comes out as 40592074.02 instead of 26.7; a measure that sums the axis on which the corruption lives cannot see that corruption. ## Next Step For three lessons the pipeline was treated as if it started **where the data arrives**. The source came about with the `generate_source` call, and the corruption was applied to it from outside. **Where the data comes from** was never asked. Yet the `unit_changed` in the last table of this lesson happens through the change of a converter in the field, and which source class can carry such an accident and which cannot is knowable in advance. A pipeline reading from an application database and a pipeline reading from a log stream or an external service are not exposed to the same corruption classes, and do not catch the same corruption at the same stage. The next lesson separates the sources by their **kind** and prints two numbers for each source class: which corruption classes it carries and the **detection lag** of each.