[SPARK-9858][SPARK-9859][SPARK-9861][SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins #9276

yhuai · 2015-10-26T05:58:26Z

https://issues.apache.org/jira/browse/SPARK-9858
https://issues.apache.org/jira/browse/SPARK-9859
https://issues.apache.org/jira/browse/SPARK-9861

yhuai · 2015-10-26T05:59:02Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala

This change is for testing purpose.

SparkQA · 2015-10-26T06:37:15Z

Test build #44342 has finished for PR 9276 at commit 3abf78f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-10-26T22:01:01Z

sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala

", in bytes" ?

SparkQA · 2015-10-28T01:44:42Z

Test build #44472 has finished for PR 9276 at commit e31ed6b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-10-28T04:41:49Z

Test build #44487 has finished for PR 9276 at commit 27af247.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

yhuai · 2015-10-28T21:57:12Z

yhuai@24e1caf is only for testing purpose. We will remove it before we commit the code.

yhuai · 2015-10-28T21:58:40Z

@JoshRosen I changed InnerJoinSuite to make our physical join operator do not use the same SparkPlan as its both left and right children (yhuai@c2463da).

SparkQA · 2015-10-28T22:26:42Z

Test build #44544 has finished for PR 9276 at commit 24e1caf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

nongli · 2015-10-28T23:42:06Z

sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala

Can you comment why that is required briefly?

SparkQA · 2015-10-29T06:14:08Z

Test build #44561 has finished for PR 9276 at commit 2130068.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

yhuai · 2015-10-29T20:06:41Z

The last three commits are just for testing purpose.

yhuai · 2015-10-29T20:27:15Z

test this please

SparkQA · 2015-10-29T21:01:12Z

Test build #44632 has finished for PR 9276 at commit f97ea39.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

SparkQA · 2015-10-29T23:03:55Z

Test build #44638 has finished for PR 9276 at commit 197b63a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * class HasSolver(Params):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

yhuai · 2015-10-30T05:40:38Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExchangeCoordinator.scala

Will add a toString method to this class.

yhuai · 2015-11-01T02:29:03Z

test this please

SparkQA · 2015-11-01T04:47:23Z

Test build #44748 has finished for PR 9276 at commit 2d1f262.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

SparkQA · 2015-11-02T01:29:04Z

Test build #44781 has finished for PR 9276 at commit 890521e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

JoshRosen · 2015-11-03T06:41:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExchangeCoordinator.scala

Add an @GuardedBy("this")?

Should this be a set instead of an array buffer in order to guard against double-registration?

If that is the case, I guess we should expose that and fix the double registration problem, right? In doEstimationIfNecessary, we have an assert assert(exchanges.length == numExchanges).

JoshRosen · 2015-11-03T07:57:00Z

Let's merge this now and post-hoc review in more detail later.

SparkQA · 2015-11-03T07:58:48Z

Test build #44882 has finished for PR 9276 at commit 60e371e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class Exchange(\n * class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: Array[Int])\n

JoshRosen · 2015-11-03T08:00:00Z

sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala

Should this call super.beforeAll()?

Seems we are fine since SparkFunSuite does not extend BeforeAndAfterAll.

yhuai · 2015-11-03T08:11:22Z

ok. I am merging it now. Will have a follow-up pr to address @JoshRosen's comments.

…to estimate the number of post-shuffle partitions for aggregates and joins https://issues.apache.org/jira/browse/SPARK-9858 https://issues.apache.org/jira/browse/SPARK-9859 https://issues.apache.org/jira/browse/SPARK-9861 Author: Yin Huai <[email protected]> Closes apache#9276 from yhuai/numReducer.

…f post-shuffle partitions for aggregates and joins (follow-up) https://issues.apache.org/jira/browse/SPARK-9858 This PR is the follow-up work of #9276. It addresses JoshRosen's comments. Author: Yin Huai <[email protected]> Closes #9453 from yhuai/numReducer-followUp. (cherry picked from commit 8211aab) Signed-off-by: Yin Huai <[email protected]>

…f post-shuffle partitions for aggregates and joins (follow-up) https://issues.apache.org/jira/browse/SPARK-9858 This PR is the follow-up work of #9276. It addresses JoshRosen's comments. Author: Yin Huai <[email protected]> Closes #9453 from yhuai/numReducer-followUp.

…to estimate the number of post-shuffle partitions for aggregates and joins https://issues.apache.org/jira/browse/SPARK-9858 https://issues.apache.org/jira/browse/SPARK-9859 https://issues.apache.org/jira/browse/SPARK-9861 Author: Yin Huai <[email protected]> Closes apache#9276 from yhuai/numReducer.

…f post-shuffle partitions for aggregates and joins (follow-up) https://issues.apache.org/jira/browse/SPARK-9858 This PR is the follow-up work of apache#9276. It addresses JoshRosen's comments. Author: Yin Huai <[email protected]> Closes apache#9453 from yhuai/numReducer-followUp.

…f post-shuffle partitions for aggregates and joins (follow-up) https://issues.apache.org/jira/browse/SPARK-9858 This PR is the follow-up work of apache/spark#9276. It addresses JoshRosen's comments. Author: Yin Huai <[email protected]> Closes #9453 from yhuai/numReducer-followUp.

cloud-fan · 2016-01-11T02:30:37Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExchangeCoordinator.scala

According to the implementation, we can't guarantee to satisfy minNumPostShufflePartitions right?

Yes, you are right. It is advisory.

darabos · 2016-03-02T13:05:21Z

Is spark.sql.adaptive.enabled documented somewhere? It's not in http://spark.apache.org/docs/1.6.0/configuration.html.

yhuai · 2016-03-02T21:28:32Z

No it is not documented. Right now, it is mainly for people who are interested in experimenting it. There are still work that needs to be done to make it support more cases.

dreamworks007 · 2016-12-01T06:41:50Z

@yhuai , could you please let us know is there any known issues / limitation with this feature ? Has this feature been tested under some large jobs ?

We are also considering automatical determining shuffle partitions, and happened to see this PR, and therefore interested in exploring this feature a little bit to see if we could productionize it for all jobs (by default).

yhuai reviewed Oct 26, 2015
View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala Outdated

Copy link

Contributor Author

yhuai Oct 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is for testing purpose.

JoshRosen reviewed Oct 26, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala Outdated

Copy link

Contributor

JoshRosen Oct 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

", in bytes" ?

nongli reviewed Oct 28, 2015
View reviewed changes

yhuai reviewed Oct 30, 2015
View reviewed changes

Make ShuffledRowRDD support fetching multiple parent partitions.

10e225c

JoshRosen reviewed Nov 3, 2015
View reviewed changes

asfgit closed this in d728d5c Nov 3, 2015

yhuai mentioned this pull request Nov 3, 2015

[SPARK-9858] [SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins (follow-up) #9453

Closed

cloud-fan reviewed Jan 11, 2016
View reviewed changes

[SPARK-9858][SPARK-9859][SPARK-9861][SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins #9276

[SPARK-9858][SPARK-9859][SPARK-9861][SQL] Add an ExchangeCoordinator to estimate the number of post-shuffle partitions for aggregates and joins #9276

Uh oh!

Conversation

yhuai commented Oct 26, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 26, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 28, 2015

Uh oh!

SparkQA commented Oct 28, 2015

Uh oh!

yhuai commented Oct 28, 2015

Uh oh!

yhuai commented Oct 28, 2015

Uh oh!

SparkQA commented Oct 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 29, 2015

Uh oh!

yhuai commented Oct 29, 2015

Uh oh!

yhuai commented Oct 29, 2015

Uh oh!

SparkQA commented Oct 29, 2015

Uh oh!

SparkQA commented Oct 29, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhuai commented Nov 1, 2015

Uh oh!

SparkQA commented Nov 1, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshRosen commented Nov 3, 2015

Uh oh!

SparkQA commented Nov 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhuai commented Nov 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darabos commented Mar 2, 2016

Uh oh!

yhuai commented Mar 2, 2016

Uh oh!

dreamworks007 commented Dec 1, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants