Class DataFrameNaFunctions

Functionality for working with missing data in DataFrames.

Index

Constructors

Methods

drop fill replace

Constructors

constructor

new DataFrameNaFunctions(df: DataFrame): DataFrameNaFunctions
Parameters
- df: DataFrame
Returns DataFrameNaFunctions
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:24

Methods

drop

drop(how?: "any" | "all", cols?: string[]): DataFrame
Returns a new DataFrame that drops rows containing null or NaN values.
Parameters
- Optionalhow: "any" | "all"
  'any': drop a row if it contains any null or NaN values. 'all': drop a row only if all its values are null or NaN.
- Optionalcols: string[]
  Optional list of column names to consider. If not specified, all columns are considered.
Returns DataFrame
DataFrame with null/NaN rows dropped
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:36
drop(cols: string[]): DataFrame
Returns a new DataFrame that drops rows containing null or NaN values in the specified columns.
Parameters
- cols: string[]
  List of column names to consider.
Returns DataFrame
DataFrame with null/NaN rows dropped
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:45
drop(minNonNulls: number, cols?: string[]): DataFrame
Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values.
Parameters
- minNonNulls: number
  Minimum number of non-null and non-NaN values required to keep a row.
- Optionalcols: string[]
  Optional list of column names to consider.
Returns DataFrame
DataFrame with rows dropped based on threshold
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:55

fill

fill(value: string | number | boolean, cols?: string[]): DataFrame
Returns a new DataFrame that replaces null or NaN values.
Parameters
- value: string | number | boolean
  Value to replace null/NaN values with. Must be a number, string, or boolean.
- Optionalcols: string[]
  Optional list of column names to consider. If not specified, all compatible columns are considered.
Returns DataFrame
DataFrame with null values replaced
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:91
fill(valueMap: { [key: string]: string | number | boolean }): DataFrame
Returns a new DataFrame that replaces null or NaN values using a map of column names to values.
Parameters
- valueMap: { [key: string]: string | number | boolean }
  Map of column names to replacement values.
Returns DataFrame
DataFrame with null values replaced
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:99

replace

replace(
    from: string | number | boolean,
    to: string | number | boolean,
    cols?: string[],
): DataFrame
Returns a new DataFrame replacing a value with another value.
Parameters
- from: string | number | boolean
  Value to be replaced. Must be a number, string, or boolean.
- to: string | number | boolean
  Value to replace with. Must be a number, string, or boolean.
- Optionalcols: string[]
  Optional list of column names to consider. If not specified, all compatible columns are considered.
Returns DataFrame
DataFrame with values replaced
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:132
replace(
replacementMap: { [key: string]: null | string | number | boolean },
cols?: string[],
): DataFrame
Returns a new DataFrame replacing values using a map.
Parameters
- replacementMap: { [key: string]: null | string | number | boolean }
  Map of old values to new replacement values.
- Optionalcols: string[]
  Optional list of column names to consider. If not specified, all compatible columns are considered.
Returns DataFrame
DataFrame with values replaced
- Defined in src/org/apache/spark/sql/DataFrameNaFunctions.ts:142