Internals on Kent Yao https://yaooqinn.github.io/tags/internals/ Recent content in Internals on Kent Yao Hugo -- 0.157.0 en-us Thu, 02 Apr 2026 00:00:00 +0000 Deep Dive into Spark SQL Metrics (Part 5): Gluten Metrics Internals https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Thu, 02 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Part 5 of the SQL Metrics deep dive. How Gluten maps Substrait plan nodes to Velox operators, aggregates metrics across pipelines, walks the MetricsUpdaterTree, and handles aggregation sub-phases and shuffle metrics. Deep Dive into Spark SQL Metrics (Part 2): Internals and How AQE Uses Them https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Part 2 of the SQL Metrics deep dive. How metrics flow from tasks to driver, and how Adaptive Query Execution uses shuffle statistics to rewrite plans at runtime.