Unit 01 of 03
Data Ingestion
Counting how data is ingested from the source into the pipeline and what each method misses: the rows transported by full load, the late-arriving change that incremental load misses, and the case where change data capture sees both; the calls a connector completes under pagination and a rate limit and the requests a retry adds; the work that topic and partition counts distribute to a consumer group and the order preserved within each partition; the result a windowing decision changes for a late-arriving record and the cost of carrying state; the 19 rows that at-most-once delivery loses, shifting every region, against the 19 duplicates of at-least-once delivery that never change the serving thanks to a keyed write; the loop that writing processed data back to an operational system opens and the number of stages after which that loop returns to its own input.
Published Updated
Lessons
- 01 Ingestion Types Counting side by side what three ingestion methods transport and what each misses: a graph built from four tasks per period carries 48 nodes and 47 edges, full load transports 15,560 rows over twelve runs and misses no change, incremental load transports 2,360 rows and misses 148 changes, and change data capture transports 2,508 rows and misses none. The 40 late-arriving rows and 108 uncorrected rows that incremental load misses pull the serving down in every region, yet the row count check stays silent at a ratio of 0.0167 and only the distribution check catches the corruption, at 0.3469 in stage 4. 9 min read
- 02 Connectors and API Consumption Counting the calls a connector completes and the rows it receives under pagination and a rate limit: with a quota of eight calls per window, on a 25-row page 83 of 96 calls complete and 2075 rows are received, on a 100-row page 20 of 24 calls complete and 2000 rows are received, so the rows missed per dropped call grow from 25 to 100 with the page size. Retry eats the quota on the small page and drops the rows received from 2075 to 2025, delays 112 pages by one window, and the 2026-12 period is never received; in return, the serving deviation falls from −374 to −125, because the loss stops being scattered and collects at the tail of the queue. The connector's own call ledger reports the shortfall without delay: expected 96, completed 83. 12 min read
- 03 Messaging Systems Counting side by side the work that the partition count distributes to a consumer group and the order preserved per partition: in a group of four consumers a single partition leaves three consumers idle, four partitions put all of them to work. In a topic partitioned by the subscriber key no subscriber is split and consumption stays at 2200 rows at every partition count, but the load skews between 708 and 492. The most balanced key, the record key, squeezes the load into 583 to 621 and splits all 200 subscribers: consumption falls to 527 rows at four partitions and the serving shifts between 616 and 977 in every region. The period key, on the other hand, loses 1,200 of the 2200 rows at two partitions while moving the serving only between −15.29 and +16.15. 13 min read
- 04 Stream Processing Turning the window decision and the carried state into numbers: when the period window is closed the moment the period ends, 40 records that arrive one period late are dropped and 10 of the 55 summary cells are polluted; when the window is kept open one period longer the dropped rows fall to 0, but the 2026-07 cell is written in 2026-08 and the carried state grows from 400 entries to 600. Tolerance is a bet on delay: when the same records are two periods late, tolerance 1 gives exactly the same result as tolerance 0. In none of the nine setups do the five served values change, only the number of polluted cells does. If the window never closes the state grows to 2400 and the mode turns back into batch processing; with the state lost without a checkpoint, 200 rows cannot be computed, the summary falls to 50 cells, and the serving shifts in every region. 14 min read
- 05 Delivery Guarantees Turning the three delivery modes into numbers on the same pipeline: with at-most-once delivery 19 rows of period 2026-07 are lost, consumption falls from 2200 to 2162, 10 of the 55 summary cells are polluted, and the serving shifts in every region (−9.18 … −28.36); with at-least-once delivery 19 rows arrive twice but consumption stays at 2200 and the serving difference comes out empty, because the dedupe stage swallows the duplication; with exactly-once delivery the keyed write zeroes out both. The real finding is in the check column: the uniqueness check catches the duplication at stage 0, while the loss slips past every check — the row ratio of 0.0079 stays below the 0.05 threshold and the largest cell deviation of 0.2488 stays below the 0.25 threshold. Loss is silent, duplication is repairable. 15 min read
- 06 Reverse ETL Counting the circle that opens when processed data is written back to an operational system: when the regional threshold computed in the serving layer is written back, the chain returns to its own input after seven stages through ingest, screen, dedupe, consumption, summary, serve and write_back. What the written-back value does in the next run is measured: the threshold written by the first run clips 1129 readings in the second run, the serving shifts in every run (the largest difference between runs is 47.00, then 10.90 and 14.36), and the loop stops at the fourth run. Where it stops is not the baseline: while the consumption rows stay at 2200, 55 of the 55 summary cells are polluted, the serving deviates by −19.27 … −59.46, and because the largest cell deviation of 0.1788 stays below the 0.25 threshold, no check fires. When the written-back value is kept apart from the pipeline's input, every run gives the same result. 14 min read