Metrics on Kent Yao https://yaooqinn.github.io/tags/metrics/ Recent content in Metrics on Kent Yao Hugo -- 0.157.0 en-us Thu, 02 Apr 2026 00:00:00 +0000 Deep Dive into Spark SQL Metrics (Part 5): Gluten Metrics Internals https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Thu, 02 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part5-gluten-internals/ Part 5 of the SQL Metrics deep dive. How Gluten maps Substrait plan nodes to Velox operators, aggregates metrics across pipelines, walks the MetricsUpdaterTree, and handles aggregation sub-phases and shuffle metrics. Deep Dive into Spark SQL Metrics (Part 6): Metrics In Action — TPC-DS q99 with Gluten https://yaooqinn.github.io/posts/spark/sql-metrics-part6-in-action/ Thu, 02 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part6-in-action/ Part 6 of the SQL Metrics series. A real-world walkthrough of TPC-DS q99 at SF10000 with Gluten/Velox, reading every metric to understand what happened during execution. Deep Dive into Spark SQL Metrics (Part 1): Types, Full Reference, and What They Mean https://yaooqinn.github.io/posts/spark/understanding-sql-metrics/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/understanding-sql-metrics/ Part 1 of a 3-part deep dive into Apache Spark’s SQL metrics system. Covers the 5 metric types, a complete reference of 100+ metrics across all operators, and how to read the numbers in the Spark UI. Deep Dive into Spark SQL Metrics (Part 2): Internals and How AQE Uses Them https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part2-internals/ Part 2 of the SQL Metrics deep dive. How metrics flow from tasks to driver, and how Adaptive Query Execution uses shuffle statistics to rewrite plans at runtime. Deep Dive into Spark SQL Metrics (Part 3): Extension APIs, UI, and REST API https://yaooqinn.github.io/posts/spark/sql-metrics-part3-extension-api/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part3-extension-api/ Part 3 of the SQL Metrics deep dive. How to extend Spark with custom metrics via the DataSource V2 API, how the UI renders them, and how to query metrics programmatically. Deep Dive into Spark SQL Metrics (Part 4): How Gluten Extends the Metrics System https://yaooqinn.github.io/posts/spark/sql-metrics-part4-gluten/ Wed, 01 Apr 2026 00:00:00 +0000 https://yaooqinn.github.io/posts/spark/sql-metrics-part4-gluten/ Part 4 of the SQL Metrics deep dive. How Apache Gluten bridges native Velox/ClickHouse metrics back to Spark’s SQL Metrics framework, adding 60+ metrics that vanilla Spark doesn’t have.