Skip to content

[SPARK-28612][SQL] Add DataFrameWriterV2 API #25354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdblue wants to merge 10 commits into from
Closed

[SPARK-28612][SQL] Add DataFrameWriterV2 API #25354

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
358e5bf
SPARK-28612: Add DataFrameWriterV2 API.
rdblue Aug 4, 2019
5dbc850
Fix bad import.
rdblue Aug 4, 2019
002339b
Fix DataFrameWriterV2 javadoc.
rdblue Aug 5, 2019
6a52509
Fix comments from reviewers.
rdblue Aug 7, 2019
f545692
Update tests for recent changes in master.
rdblue Aug 20, 2019
6c0a98b
Improve error message in bucket function.
rdblue Aug 21, 2019
1bc6954
Move partitioning functions into a partitioning object.
rdblue Aug 22, 2019
e424c2c
Remove asFormat methods.
rdblue Aug 30, 2019
57e6c5b
Update tests for InMemoryTableCatalog consolidation.
rdblue Aug 30, 2019
9864d42
Update test cases for CTAS with nullable schemas.
rdblue Aug 31, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix comments from reviewers.
  • Loading branch information
@rdblue
rdblue committed Aug 30, 2019
commit 6a5250932acfb4e247aaee6ab1b6940a7cc3b377
2 changes: 1 addition & 1 deletion ...atalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/PartitionTransforms.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import org.apache.spark.sql.types.{DataType, IntegerType}
* These expressions are used to pass transformations from the DataFrame API:
*
* {{{
* df.writeTo("catalog.db.table").partitionBy($"category", days($"timestamp")).create()
* df.writeTo("catalog.db.table").partitionedBy($"category", days($"timestamp")).create()
* }}}
*/
abstract class PartitionTransformExpression extends Expression with Unevaluable {
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3213,7 +3213,7 @@ class Dataset[T] private[sql](
// TODO: streaming could be adapted to use this interface
if (isStreaming) {
logicalPlan.failAnalysis(
"'write' can not be called on streaming Dataset/DataFrame")
"'writeTo' can not be called on streaming Dataset/DataFrame")
Copy link
Contributor

@brkyvz brkyvz Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be good to add: use 'writeStream' instead

Copy link
Contributor Author

@rdblue rdblue Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't include this because I'd rather not have the v2 API recommend using the v1 API. That seems confusing to me.

}
new DataFrameWriterV2[T](table, this)
}
Expand Down
1 change: 1 addition & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/functions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ import org.apache.spark.util.Utils
* @groupname window_funcs Window functions
* @groupname string_funcs String functions
* @groupname collection_funcs Collection functions
* @groupname partition_transforms Partition transform functions
* @groupname Ungrouped Support functions for DataFrames
* @since 1.3.0
*/
Expand Down