Skip to content

[SPARK-22614] Dataset API: repartitionByRange(...) #19828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adrian-ionescu wants to merge 12 commits into from
Closed

[SPARK-22614] Dataset API: repartitionByRange(...) #19828

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
08527b0
repartitionByRange() + repartition() with SortOrders
adrian-ionescu Nov 27, 2017
950b3dc
avoid changing semantics for .repartition()
adrian-ionescu Nov 27, 2017
671e9e4
review feedback: val partition = ... + unit test
adrian-ionescu Nov 27, 2017
fe690fc
move test to DataFrameSuite + minor ws
adrian-ionescu Nov 28, 2017
66b192d
handle empty partition-by expr list as RoundRobin
adrian-ionescu Nov 28, 2017
012d617
update docs
adrian-ionescu Nov 29, 2017
60ec0e3
fix typo: hash -> range
adrian-ionescu Nov 29, 2017
e0bda1d
Revert "fix typo: hash -> range"
adrian-ionescu Nov 30, 2017
2f0f8f4
Revert "update docs"
adrian-ionescu Nov 30, 2017
f6cd388
Revert "handle empty partition-by expr list as RoundRobin"
adrian-ionescu Nov 30, 2017
67ca139
avoid changing repartition() and make repartitionByRange() throw on e…
adrian-ionescu Nov 30, 2017
012baa0
fix triple quotes
adrian-ionescu Nov 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update docs
  • Loading branch information
@adrian-ionescu
adrian-ionescu committed Nov 29, 2017
commit 012d617430befd8b028562c4d8f7b49cf8776659
8 changes: 4 additions & 4 deletions ...st/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -846,10 +846,10 @@ case class RepartitionByExpression(
s"${getClass.getSimpleName} expects that either all its `partitionExpressions` are of type " +
"`SortOrder`, which means `RangePartitioning`, or none of them are `SortOrder`, which " +
"means `HashPartitioning`. In this case we have:" +
s""""
|SortOrder: ${sortOrder}
|NonSortOrder: ${nonSortOrder}
""".stripMargin)
s"""
|SortOrder: ${sortOrder}
|NonSortOrder: ${nonSortOrder}
""".stripMargin)

if (sortOrder.nonEmpty) {
RangePartitioning(sortOrder.map(_.asInstanceOf[SortOrder]), numPartitions)
Expand Down
20 changes: 10 additions & 10 deletions sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2723,8 +2723,8 @@ class Dataset[T] private[sql](
}

/**
* Returns a new Dataset partitioned by the given partitioning expressions into
* `numPartitions`. The resulting Dataset is hash partitioned.
* Returns a new Dataset that is hash partitioned by the given expressions into `numPartitions`.
* If no expressions are specified, round robin partitioning is used.
*
* This is the same operation as "DISTRIBUTE BY" in SQL (Hive QL).
*
Expand All @@ -2745,9 +2745,9 @@ class Dataset[T] private[sql](
}

/**
* Returns a new Dataset partitioned by the given partitioning expressions, using
* `spark.sql.shuffle.partitions` as number of partitions.
* The resulting Dataset is hash partitioned.
* Returns a new Dataset that is hash partitioned by the given expressions, using
* `spark.sql.shuffle.partitions` as the number of partitions. If no expressions are specified,
* round robin partitioning is used.
*
* This is the same operation as "DISTRIBUTE BY" in SQL (Hive QL).
*
Expand All @@ -2760,8 +2760,8 @@ class Dataset[T] private[sql](
}

/**
* Returns a new Dataset partitioned by the given partitioning expressions into
* `numPartitions`. The resulting Dataset is range partitioned.
* Returns a new Dataset that is hash partitioned by the given expressions into `numPartitions`.
Copy link
Member

@gatorsmile gatorsmile Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hash -> range

* If no expressions are specified, round robin partitioning is used.
*
* @group typedrel
* @since 2.3.0
Expand All @@ -2780,9 +2780,9 @@ class Dataset[T] private[sql](
}

/**
* Returns a new Dataset partitioned by the given partitioning expressions, using
* `spark.sql.shuffle.partitions` as number of partitions.
* The resulting Dataset is range partitioned.
* Returns a new Dataset that is range partitioned by the given expressions, using
* `spark.sql.shuffle.partitions` as the number of partitions. If no expressions are specified,
* round robin partitioning is used.
*
* @group typedrel
* @since 2.3.0
Expand Down