https://yaooqinn.github.io/tags/pipelines/ Recent content in Pipelines on Kent Yao Hugo -- 0.157.0 en-us Sat, 28 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/spark-declarative-pipelines/ Sat, 28 Mar 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/spark-declarative-pipelines/ Apache Spark 4.1 introduces Spark Declarative Pipelines (SDP) — a declarative framework that lets you define what your data should look like, not how to compute it. As a Spark PMC Member, here’s my take on what this means for data engineering.