Metrics

Deep Dive into Spark SQL Metrics (Part 5): Gluten Metrics Internals

Part 5 of the SQL Metrics deep dive. How Gluten maps Substrait plan nodes to Velox operators, aggregates metrics across pipelines, walks the MetricsUpdaterTree, and handles aggregation sub-phases and shuffle metrics.

Deep Dive into Spark SQL Metrics (Part 6): Metrics In Action — TPC-DS q99 with Gluten

Part 6 of the SQL Metrics series. A real-world walkthrough of TPC-DS q99 at SF10000 with Gluten/Velox, reading every metric to understand what happened during execution.

Deep Dive into Spark SQL Metrics (Part 1): Types, Full Reference, and What They Mean

Part 1 of a 3-part deep dive into Apache Spark’s SQL metrics system. Covers the 5 metric types, a complete reference of 100+ metrics across all operators, and how to read the numbers in the Spark UI.

Deep Dive into Spark SQL Metrics (Part 2): Internals and How AQE Uses Them

Part 2 of the SQL Metrics deep dive. How metrics flow from tasks to driver, and how Adaptive Query Execution uses shuffle statistics to rewrite plans at runtime.

Deep Dive into Spark SQL Metrics (Part 3): Extension APIs, UI, and REST API

Part 3 of the SQL Metrics deep dive. How to extend Spark with custom metrics via the DataSource V2 API, how the UI renders them, and how to query metrics programmatically.

Deep Dive into Spark SQL Metrics (Part 4): How Gluten Extends the Metrics System

Part 4 of the SQL Metrics deep dive. How Apache Gluten bridges native Velox/ClickHouse metrics back to Spark’s SQL Metrics framework, adding 60+ metrics that vanilla Spark doesn’t have.