Curriculum

Data Engineering and MLOps

Measuring a pipeline that reports when it crashes but says nothing when it carries wrong data: a data pipeline's number is not the rows it carries, it is how many steps later a wrong row is caught. Turning the stages of the lifecycle and the quality dimensions into measurable checks, the effect of the storage layout on query cost across the warehouse, lake and lakehouse approaches, the rerunnability of ingestion, transformation and orchestration, partitioning and performance in distributed processing, the reproducibility and deployment of model training, and the governance of a model in production with drift monitoring. In every lesson the unchecked pipeline, the checked pipeline and the cost of the check itself stand side by side.

Measuring a pipeline that reports when it crashes but says nothing when it carries wrong data: a data pipeline's number is not the rows it carries, it is how many steps later a wrong row is caught. Turning the stages of the lifecycle and the quality dimensions into measurable checks, the effect of the storage layout on query cost across the warehouse, lake and lakehouse approaches, the rerunnability of ingestion, transformation and orchestration, partitioning and performance in distributed processing, the reproducibility and deployment of model training, and the governance of a model in production with drift monitoring. In every lesson the unchecked pipeline, the checked pipeline and the cost of the check itself stand side by side.