Skip to content

[SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations #39131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
EnricoMi wants to merge 5 commits into from
Closed

[SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations #39131

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
eebeb34
Use canPushThroughCondition for LeftSemi and LeftAnti joins
EnricoMi Dec 19, 2022
76dcd1e
Fix LeftSemiPushdownSuite class name (match file name)
EnricoMi Dec 19, 2022
ade61b8
Rename LeftSemiAnti to LeftSemi in LeftSemiAntiJoinPushDownSuite
EnricoMi Dec 19, 2022
cc28b21
Adjust test names to one naming scheme
EnricoMi Dec 19, 2022
2ae3a41
Add test to LeftSemiAntiJoinPushDownSuite
EnricoMi Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Add test to LeftSemiAntiJoinPushDownSuite
  • Loading branch information
@EnricoMi
EnricoMi committed Dec 19, 2022
commit 2ae3a415c18ff61ce1324e8c2cc90b0757622645
19 changes: 19 additions & 0 deletions ...rc/test/scala/org/apache/spark/sql/catalyst/optimizer/LeftSemiAntiJoinPushDownSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,25 @@ class LeftSemiAntiJoinPushDownSuite extends PlanTest {
}
}

Seq(LeftSemi, LeftAnti).foreach { case jt =>
test(s"Aggregate: $jt join no pushdown - join condition refers left leg and right leg child") {
val aggregation = testRelation
.select($"b".as("id"), $"c")
.groupBy($"id")($"id", sum($"c").as("sum"))

// reference "b" exists in left leg, and the children of the right leg of the join
val originalQuery = aggregation.select(($"id" + 1).as("id_plus_1"), $"sum")
.join(aggregation, joinType = jt, condition = Some($"id" === $"id_plus_1"))
val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer = testRelation
.select($"b".as("id"), $"c")
.groupBy($"id")(($"id" + 1).as("id_plus_1"), sum($"c").as("sum"))
.join(aggregation, joinType = jt, condition = Some($"id" === $"id_plus_1"))
.analyze
comparePlans(optimized, correctAnswer)
}
}

Seq(LeftSemi, LeftAnti).foreach { case outerJT =>
Seq(Inner, LeftOuter, RightOuter, Cross).foreach { case innerJT =>
test(s"$outerJT no pushdown - join condition refers none of the leg - join type $innerJT") {
Expand Down