Data Engineering and Data Science Distinction

The previous two lessons read the pipeline from a single eye. In production there is no single eye around this pipeline. There is a role that writes the stages, looks at the trace table and is alerted when a row drops; there is another role that takes the table at the end of the pipeline as a feature table, builds a rule or a model on it and looks at the performance number. The two stand at different places of the same pipeline.

The role distinction itself is not the subject of this curriculum; the boundaries of responsibility were drawn separately in the Introduction to Data Analytics, Introduction to Machine Learning and Introduction to AI Engineering courses. The only thing this lesson adds is seating the distinction on the stages of the pipeline: from which stage the number of which role is read, and who measures the stage that remains between the two numbers. When this question is asked, an unpleasant result appears: the same corruption reflects differently on the two measures, the two rank the six classes differently, and one class leaves no trace in either of them.

• DL16. The pipeline, the source, the six corruption classes and the seed are the same as in the previous two lessons.
• DL17. The procedure for role definitions is not repeated; it was measured in the three introductory courses above. Here the distinction is written only as the stage from which the measure is read.
• DL18. The measure of the data engineer is the row trace: the remaining counts of the five stages. This is the same as the table in the first lesson.
• DL19. The measure of the model builder is the performance: the mean absolute deviation, in m3, of a forecast rule that takes the per-region mean served by the pipeline as its feature. The rule divides the served value of a region by the subscriber count of that region and gives this as the forecast to every subscriber in the region.
• DL20. The test set is the periods 2026-11 and 2026-12 (400 rows). These two periods are not corrupted in any of the six classes; so every difference in the deviation comes from the model’s feature side, not from the test side.
• DL21. The model builder takes the feature table from the pipeline and does not reach the source. This is the boundary in production: the served layer is the contract between the two roles.
• DL22. The procedure for feature engineering and model evaluation is not repeated; it was measured in the Data Preparation and Feature Engineering and Model Evaluation, Interpretability and Ethics courses. The model is deliberately at its simplest here. The aim is not for the model to be good, but what its measure is sensitive to.
• DL23. No check; the baseline is the uninstrumented pipeline, the corruption caught is 0, the lag is undefined. The pipeline is deterministic, a second run gives the same table. Time is not written.

Two Measures, the Same Pipeline

The block below builds the pipeline and the six corruption classes as they are, adds the model builder’s measure on top, and prints two numbers side by side for each class: is the row trace the same as the intact pipeline, is the mean absolute deviation the same as the intact pipeline.

# FICTIONAL pipeline, FICTIONAL corruption classes and a FICTIONAL forecast rule.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


# region values stay Turkish: the summary table below is printed in sorted order
# dogu/bati/kuzey/guney/merkez = east/west/north/south/central
REGION = ["dogu", "bati", "kuzey", "guney", "merkez"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


CORRUPTIONS = ["late_arrival", "duplicated", "schema_changed", "unit_changed", "missing_partition", "type_drift"]


def corrupt(records, cls, period="2026-07"):
    """Corrupts a single period , the remaining periods stay intact."""
    out, counter = [], 0
    for k in records:
        if k["period"] != period:
            out.append(k)
            continue
        if cls == "late_arrival":
            if k["subscriber"] <= "A040":
                counter += 1
                continue
            out.append(k)
        elif cls == "duplicated":
            out.append(k)
            if k["subscriber"] <= "A060":
                out.append(dict(k))
                counter += 1
        elif cls == "schema_changed":
            y = dict(k)
            y["index_value"] = y.pop("index")
            counter += 1
            out.append(y)
        elif cls == "unit_changed":
            y = dict(k)
            y["index"] = k["index"] * 1000          # liters instead of m3
            y["unit"] = "m3"                        # LABEL DID NOT CHANGE
            counter += 1
            out.append(y)
        elif cls == "missing_partition":
            if k["region"] == "kuzey":
                counter += 1
                continue
            out.append(k)
        elif cls == "type_drift":
            y = dict(k)
            y["index"] = str(k["index"])
            counter += 1
            out.append(y)
    return out, counter


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


def s4_summary(records):
    totals = {}
    for k in records:
        key = (k["region"], k["period"])
        totals[key] = totals.get(key, 0) + k["m3"]
    return totals, 0


def s5_serve(summary):
    region = {}
    for (b, d), v in summary.items():
        region.setdefault(b, []).append(v)
    return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0


STAGES = [("screen", s1_screen), ("dedupe", s2_dedupe), ("consumption", s3_consumption),
          ("summary", s4_summary), ("serve", s5_serve)]


def pipeline(records):
    trace, data = [], records
    for name, f in STAGES:
        data, dropped = f(data)
        n = len(data) if hasattr(data, "__len__") else 0
        trace.append({"stage": name, "remaining": n, "dropped": dropped})
    return data, trace


TEST_PERIODS = ["2026-11", "2026-12"]


def model_metric(records):
    """Model builder , takes the per-region mean served by the pipeline as a feature."""
    v1, _ = s1_screen(records)
    v2, _ = s2_dedupe(v1)
    t, _ = s3_consumption(v2)
    summ, _ = s4_summary(t)
    features, _ = s5_serve(summ)
    n = {}
    for k in t:
        n.setdefault(k["region"], set()).add(k["subscriber"])
    forecast = {b: features[b] / len(n[b]) for b in features}
    test = [k for k in t if k["period"] in TEST_PERIODS]
    return round(sum(abs(k["m3"] - forecast[k["region"]]) for k in test) / len(test), 3), len(test)


source = generate_source()
base_trace = [a["remaining"] for a in pipeline(source)[1]]
base_dev, base_n = model_metric(source)
print("test rows", base_n, " baseline deviation", base_dev, " m3")
print(f"{'label':<19}{'row trace':>26}{'rows':>7}{'deviation':>11}{'diff':>8}{'model':>7}")
print(f"{'intact':<19}{str(base_trace):>26}{'-':>7}{base_dev:>11}{'-':>8}{'-':>7}")
for cls in CORRUPTIONS:
    trace = [a["remaining"] for a in pipeline(corrupt(source, cls)[0])[1]]
    dev, _ = model_metric(corrupt(source, cls)[0])
    print(f"{cls:<19}{str(trace):>26}{('yes' if trace == base_trace else 'no'):>7}"
          + f"{dev:>11}{round(dev - base_dev, 3):>8}"
          + f"{('yes' if dev == base_dev else 'no'):>7}")

test rows 400  baseline deviation 10.832  m3
label                               row trace   rows  deviation    diff  model
intact              [2400, 2400, 2200, 55, 5]      -     10.832       -      -
late_arrival        [2360, 2360, 2120, 55, 5]     no     10.937   0.105     no
duplicated          [2460, 2400, 2200, 55, 5]     no     10.832     0.0    yes
schema_changed      [2200, 2200, 1800, 45, 5]     no     10.835   0.003     no
unit_changed        [2400, 2400, 2200, 55, 5]    yes     10.832     0.0    yes
missing_partition   [2354, 2354, 2108, 53, 5]     no     10.829  -0.003     no
type_drift          [2400, 2400, 2200, 55, 5]    yes     10.832     0.0    yes

One Who Sees Rows and One Who Sees Performance

The two measures rank the same six classes differently, and this difference in ranking is the real number of the role distinction.

In the data engineer’s trace the noisiest class is schema_changed: the consumption output falls from 2200 to 1800, the summary from 55 to 45. The change of a field name takes away a sixth of the source, and this is a drop that cannot be overlooked on a dashboard table. The same class moves the model builder’s measure by 0.003, that is, a relative difference of 0.0003 over 10.832. When the model builder looks at this number, they think nothing happened, and they are right: even if a sixth of the training data is lost, the per-region mean comes out almost the same from the remaining nine periods.

The opposite direction is more interesting. late_arrival removes only 40 rows from the source, that is, 0.0167 of 2400 (the smallest step of the row ratio is 0.0004, so this ratio can be read to three digits); this is a loss that falls below the band of most row count checks. In return it moves the model builder’s deviation by 0.105, and among the six classes it is the one that shifts it the most. The reason is not in the table but in the definition of the pipeline: the missing rows are not randomly distributed, they are concentrated in the first forty subscribers and pulled the region means down together. A loss that is small in number but one-directional is more harmful than a loss that is large in number but balanced, and the row count does not write this distinction.

The third row is the cleanest distinction between the two roles. duplicated is plainly visible in the engineer’s trace: the screen output is 2460 instead of 2400. In the model builder’s measure, however, the difference is exactly zero, because the deduplication stage drops the sixty double records and the intact set reaches the feature table. This is a case where the pipeline does its job right; still, the engineer needs to see it, because the next double record from the same producer may not be caught at the same place.

The missing_partition row shows the most dangerous side of reading a measure. The deviation falls from 10.832 to 10.829, that is, the corruption improves the model’s measure. The improvement is not real; since two periods of region kuzey (north) are completely lost, the served mean of that region is computed from the remaining nine cells and, by chance, falls slightly closer to the test periods. The result is this: a measure not getting worse is not proof of health, because the corruption can improve the measure too. The direction of the sign says nothing about the presence of corruption.

What Is Visible in Neither

The last reading of the table is two rows: unit_changed and type_drift. In both, the row trace is exactly the same as the intact pipeline, and the deviation is exactly the same. The routine measure of both roles is completely blind to these two classes.

The reason is superficial for type_drift: the pipeline turns the text into a number, the conversion succeeds, and neither the rows nor the values change. The only thing that sees it is a type check that never looks at the value of the row, and such a check can run at the zeroth step of the pipeline. The reason is structural for unit_changed, and the block below shows it.

def region_total(records):
    v1, _ = s1_screen(records)
    v2, _ = s2_dedupe(v1)
    t, _ = s3_consumption(v2)
    summ, _ = s4_summary(t)
    total = {}
    for (b, d), v in summ.items():
        total[b] = total.get(b, 0) + v
    return total


def period_mean(records):
    v1, _ = s1_screen(records)
    v2, _ = s2_dedupe(v1)
    t, _ = s3_consumption(v2)
    d = {}
    for k in t:
        d.setdefault(k["period"], []).append(k["m3"])
    return {a: round(sum(v) / len(v), 2) for a, v in d.items()}


first = {k["subscriber"]: k["index"] for k in source if k["period"] == PERIODS[0]}
last = {k["subscriber"]: k["index"] for k in source if k["period"] == PERIODS[-1]}
s_top, b_top = region_total(source), region_total(corrupt(source, "unit_changed")[0])
print("region   ends diff  intact summary  corrupted summary")
for b in sorted(s_top):
    ends = sum(last[a] - first[a] for a, v in SUBSCRIBERS.items() if v["region"] == b)
    print(f"{b:<9}{ends:>11}{s_top[b]:>16}{b_top[b]:>19}")
s_d, b_d = period_mean(source), period_mean(corrupt(source, "unit_changed")[0])
print("period        intact      corrupted")
for d in PERIODS[5:9]:
    print(f"{d:<10}{s_d[d]:>10}{b_d[d]:>15}")

region   ends diff  intact summary  corrupted summary
bati            9164            9164               9164
dogu            9695            9695               9695
guney          11721           11721              11721
kuzey          13338           13338              13338
merkez         14536           14536              14536
period        intact      corrupted
2026-06        24.08          24.08
2026-07         26.7    40592074.02
2026-08        25.55   -40592021.77
2026-09        28.96          28.96

The three columns of the first table give the same number for every region. The left column was computed without going through the pipeline at all, directly from the end points of the source: the last period index of the region’s subscribers minus the first period index. The middle and right columns are the sum of the eleven summary cells of the pipeline, with the intact and the corrupted source. The equality of the three is not a measurement result but an identity: the sum of successive differences equals the difference of the ends. Because unit_changed does not touch the ends, the total cannot change, and the served value, which divides the total by eleven, cannot change either. This is a result known even without a run; the run has only verified it.

The second table reads the same feature table on the period axis, and there the corruption shouts: the July mean is 40592074.02 instead of 26.7, the August mean is negative 40592021.77 instead of 25.55. The data is not hiding it; what hides it is both roles summing over the period axis. The engineer sums the rows per stage, the model builder sums the values per region; the corruption lives exactly on the axis under these two sums, with two signs that cancel each other. The rule that follows from here is general and applicable: a measure that sums the axis on which the corruption lives cannot see that corruption.

The count of the role distinction can also be read from here. The engineer’s measure is read from the first, second and third stages; the model builder’s measure from the output of the fifth stage. Between them stands the fourth stage, and the routine measure of neither role reads those fifty-five cells. The place where the corruption sits is exactly there. The boundary is not a question of job definition; it is a question of which number is read from which stage of the pipeline, and as long as a stage remains unmeasured between the measures of the two roles, the boundary produces a gap.

Summary

• The boundary between data engineering and data science is not a job definition but the stage from which the measure is read: the engineer reads the row trace of the first to third stages, the model builder reads the output of the fifth stage.
• The two measures rank the same six classes differently: schema_changed is the noisiest class in the trace (2200 rows fall to 1800) yet moves the deviation by only 0.003; late_arrival removes only 40 rows yet moves the deviation by 0.105, shifting it the most.
• A loss that is small in number but one-directional is more harmful than a loss that is large in number but balanced, and the row count does not write this distinction.
• missing_partition lowers the deviation from 10.832 to 10.829: a corruption can improve the measure, so a measure not getting worse is not proof of health.
• unit_changed and type_drift give exactly the same result in both measures. The invisibility of the unit change comes from a telescoping identity: the sum of the eleven cells of a region equals the difference of the end indexes, and the corruption does not touch the ends.
• When the same table is read on the period axis, the July mean comes out as 40592074.02 instead of 26.7; a measure that sums the axis on which the corruption lives cannot see that corruption.

Next Step

For three lessons the pipeline was treated as if it started where the data arrives. The source came about with the generate_source call, and the corruption was applied to it from outside. Where the data comes from was never asked. Yet the unit_changed in the last table of this lesson happens through the change of a converter in the field, and which source class can carry such an accident and which cannot is knowable in advance. A pipeline reading from an application database and a pipeline reading from a log stream or an external service are not exposed to the same corruption classes, and do not catch the same corruption at the same stage. The next lesson separates the sources by their kind and prints two numbers for each source class: which corruption classes it carries and the detection lag of each.