Partitions the output by the given columns on the file system. If specified, the output is laid out on the file system similar to Hive's partitioning scheme. As an example, when we partition a dataset by year and then month, the directory layout would look like:
Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.
Buckets the output by the given columns. If specified, the output is laid out on the file system similar to Hive's bucketing scheme, but with a different bucket hash function and is not compatible with Hive's bucketing.
Sorts the output in each bucket by the given columns.
Clusters the output by the given columns on the storage. The rows with matching values in the specified clustering columns will be consolidated within the same group.
For instance, if you cluster a dataset by date, the data sharing the same date will be stored together in a file. This arrangement improves query efficiency when you apply selective filters to these clustering columns, thanks to data skipping.
Saves the content of the DataFrame as the specified table.
Optionalpath: stringInserts the content of the DataFrame to the specified table. It requires that the schema of
the DataFrame is the same as the schema of the table.
Saves the content of the DataFrame as the specified table.
In the case the table already exists, behavior of this function depends on the save mode,
specified by the mode function (default to throwing an exception). When mode is
Overwrite, the schema of the DataFrame does not need to be the same as that of the
existing table.
When mode is Append, if there is an existing table, we will use the format and options of
the existing table. The column order in the schema of the DataFrame doesn't need to be same
as that of the existing table. Unlike insertInto, saveAsTable will use the column names
to find the correct column positions. For example:
In this method, save mode is used to determine the behavior if the data source table exists in Spark catalog. We will always overwrite the underlying data of data source (e.g. a table in JDBC data source) if the table doesn't exist in Spark catalog, and will always append to the underlying data of data source if the table already exists.
When the DataFrame is created from a non-partitioned HadoopFsRelation with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
Saves the content of the DataFrame to an external database table via JDBC. In the case the
table already exists in the external database, behavior of this function depends on the save
mode, specified by the mode function (default to throwing an exception).
Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems.
JDBC-specific option and parameter documentation for storing tables via JDBC in Data Source Option in the version you use.
JDBC database url of the form jdbc:subprotocol:subname.
Name of the table in the external database.
JDBC database connection arguments, a list of arbitrary string tag/value. Normally at least a "user" and "password" property should be included. "batchsize" can be used to control the number of rows per insert. "isolationLevel" can be one of "NONE", "READ_COMMITTED", "READ_UNCOMMITTED", "REPEATABLE_READ", or "SERIALIZABLE", corresponding to standard transaction isolation levels defined by JDBC's Connection object, with default of "READ_UNCOMMITTED".
Saves the content of the DataFrame in JSON format ( JSON
Lines text format or newline-delimited JSON) at the specified path. This is equivalent
to:
{{{
format("json").save(path)
}}}
You can find the JSON-specific options for writing JSON files in Data Source Option in the version you use.
Saves the content of the DataFrame in Parquet format at the specified path. This is
equivalent to:
{{{
format("parquet").save(path)
}}}
Parquet-specific option(s) for writing Parquet files can be found in Data Source Option in the version you use.
Saves the content of the DataFrame in ORC format at the specified path. This is equivalent
to:
{{{
format("orc").save(path)
}}}
ORC-specific option(s) for writing ORC files can be found in Data Source Option in the version you use.
Saves the content of the DataFrame in a text file at the specified path. The DataFrame must
have only one column that is of string type. Each row becomes a new line in the output file.
You can find the text-specific options for writing text files in Data Source Option in the version you use.
Saves the content of the DataFrame in CSV format at the specified path. This is equivalent
to:
{{{
format("csv").save(path)
}}}
You can find the CSV-specific options for writing CSV files in Data Source Option in the version you use.
Saves the content of the DataFrame in XML format at the specified path. This is equivalent
to:
{{{
format("xml").save(path)
}}}
Note that writing a XML file from DataFrame having a field ArrayType with its element as
ArrayType would have an additional nested field for the element. For example, the
DataFrame having a field below,
{@code fieldA [[data1], [data2]]}
would produce a XML file below. {@code
Namely, roundtrip in writing and reading can end up in different schema structure.
You can find the XML-specific options for writing XML files in Data Source Option in the version you use.
Interface used to write a [[Dataset]] to external storage systems (e.g. file systems, key-value stores, etc). Use
DataFrame.write()to access this.Stable
Since
1.0.0
See
https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/DataFrameWriter.html
Author
Kent Yao yao@apache.org
TODO: Some features are not implemented yet: