Class RelationalGroupedDataset

constructor

• new RelationalGroupedDataset(
      df: DataFrame,
      groupingExprs: string[] | Column[],
      groupType: GroupType,
      pivotValue?: undefined | Aggregate_Pivot,
      groupingSets?: Column[][],
  ): RelationalGroupedDataset

  Parameters

  • df: DataFrame
  • groupingExprs: string[] | Column[]
  • groupType: GroupType
  • pivotValue: undefined | Aggregate_Pivot = undefined
  • groupingSets: Column[][] = []

  Returns RelationalGroupedDataset

Readonlydf

ReadonlygroupingExprs

ReadonlygroupType

ReadonlypivotValue

ReadonlygroupingSets

toDF

• toDF(...aggExprs: Column[]): DataFrame

  Parameters

  • ...aggExprs: Column[]

  Returns DataFrame

count

• count(): DataFrame

  Returns DataFrame

sum

• sum(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

avg

• avg(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

mean

• mean(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

min

• min(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

max

• max(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

first

• first(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

last

• last(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

stddev

• stddev(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

stddevPop

• stddevPop(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

stddevSamp

• stddevSamp(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

variance

• variance(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

varPop

• varPop(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

varSamp

• varSamp(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

collect_list

• collect_list(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

collect_set

• collect_set(...cols: string[]): DataFrame

  Parameters

  • ...cols: string[]

  Returns DataFrame

agg

• agg(exprs: Record<string, string>): DataFrame

  Parameters

  • exprs: Record<string, string>

  Returns DataFrame

• agg(...exprs: Column[]): DataFrame

  Parameters

  • ...exprs: Column[]

  Returns DataFrame

pivot

• pivot(pivotColumn: string | Column, values?: any[]): RelationalGroupedDataset

  Pivots a column of the current DataFrame and performs the specified aggregation.

  This method is only supported after a groupBy operation. There are two versions of pivot: one with explicit pivot values and one without.

  Parameters

  • pivotColumn: string | Column

    Column name or Column to pivot on
  • Optionalvalues: any[]

    Optional list of values that will be translated to columns in the output DataFrame

  Returns RelationalGroupedDataset

  A new RelationalGroupedDataset with pivot configuration

  Example

  // Pivot without values (Spark will compute distinct values)
  df.groupBy("year").pivot("course").sum("earnings")
  
  // Pivot with explicit values (more efficient)
  df.groupBy("year").pivot("course", ["dotNET", "Java"]).sum("earnings")
  Copy

groupMap

• groupMap(
      pythonCode: string,
      outputSchema: StructType,
      pythonVersion?: string,
  ): DataFrame

  Apply a function to each group of the DataFrame.

  This method applies a user-defined function to each group. The function receives the group key and an iterator of rows for that group, and should return an iterator of rows.

  Parameters

  • pythonCode: string

    Python code as a string defining the group processing function
  • outputSchema: StructType

    The output schema for the transformed DataFrame
  • pythonVersion: string = '3.11'

    Python version (default: '3.11')

  Returns DataFrame

  A new DataFrame with the function applied to each group

  Example

  const pythonCode = `
  def group_func(key, rows):
      total = sum(row.value for row in rows)
      yield (key.category, total)
  `;
  const schema = DataTypes.createStructType([
    DataTypes.createStructField('category', DataTypes.StringType, false),
    DataTypes.createStructField('total', DataTypes.IntegerType, false),
  ]);
  const result = df.groupBy('category').groupMap(pythonCode, schema);
  Copy