Skip to content

[SPARK-32201][SQL] More general skew join pattern matching #29021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LantaoJin wants to merge 15 commits into from
Closed

[SPARK-32201][SQL] More general skew join pattern matching #29021

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
479d56b
[SPARK-32201][SQL] More general skew join pattern matching
LantaoJin Jul 7, 2020
947927a
remove the println
LantaoJin Jul 7, 2020
607eb08
cannot split if AggExec in one side
LantaoJin Jul 7, 2020
80bef0d
add more agg exec
LantaoJin Jul 8, 2020
dd09e70
Add a new parititioning CoalescedHashPartitioning
LantaoJin Jul 10, 2020
76dc3ea
fix ut
LantaoJin Jul 11, 2020
7465bfa
more general canSplitXSide
LantaoJin Jul 15, 2020
0950e9a
fix ut
LantaoJin Jul 16, 2020
42a52a1
Merge branch 'master' into SPARK-32201
LantaoJin Jul 16, 2020
973d87e
add another skew test case
LantaoJin Jul 16, 2020
d65a210
canUseLocalShuffleReader should consider skew optimization
LantaoJin Jul 16, 2020
6f7dbc2
fix ut
LantaoJin Jul 16, 2020
aa3af93
approach 2: remove 'CoalescedHashPartitioning' and refine code
LantaoJin Jul 20, 2020
3cd411f
remove dead code
LantaoJin Jul 20, 2020
5bed68c
Merge remote-tracking branch 'upstream/master' into SPARK-32201
LantaoJin Jul 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add more agg exec
  • Loading branch information
@LantaoJin
LantaoJin committed Jul 8, 2020
commit 80bef0d2c22bf91d216784f29839b95f22fb230f
18 changes: 13 additions & 5 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.spark.{MapOutputStatistics, MapOutputTrackerMaster, SparkEnv}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.aggregate.HashAggregateExec
import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec}
import org.apache.spark.sql.execution.exchange.{EnsureRequirements, ShuffleExchangeExec}
import org.apache.spark.sql.execution.joins.SortMergeJoinExec
import org.apache.spark.sql.internal.SQLConf
Expand Down Expand Up @@ -133,13 +133,21 @@ case class OptimizeSkewedJoin(conf: SQLConf) extends Rule[SparkPlan] {

private def canSplitLeftSide(joinType: JoinType, plan: SparkPlan) = {
(joinType == Inner || joinType == Cross || joinType == LeftSemi ||
joinType == LeftAnti || joinType == LeftOuter) &&
plan.find(_.isInstanceOf[HashAggregateExec]).isEmpty
joinType == LeftAnti || joinType == LeftOuter) && !containsAggregateExec(plan)
}

private def canSplitRightSide(joinType: JoinType, plan: SparkPlan) = {
(joinType == Inner || joinType == Cross || joinType == RightOuter) &&
plan.find(_.isInstanceOf[HashAggregateExec]).isEmpty
(joinType == Inner || joinType == Cross ||
joinType == RightOuter) && !containsAggregateExec(plan)
}

private def containsAggregateExec(plan: SparkPlan) = {
plan.find {
case _: HashAggregateExec => true
case _: SortAggregateExec => true
case _: ObjectHashAggregateExec => true
case _ => false
}.isDefined
}

private def getSizeInfo(medianSize: Long, sizes: Seq[Long]): String = {
Expand Down