-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-12656] [SQL] Implement Intersect with Left-semi Join #10630
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
01e4cdf
6835704
9180687
b38a21e
d2b84af
fda8025
ac0dccd
6e0018b
0546772
b37a64f
c2a872c
ab6dbd7
4276356
0bd1771
7bd102b
bfa99c5
cd23b03
100174a
9aad1cf
6742984
e4c34f0
2dab708
9864b3f
24cea7d
27192be
a932cdb
04a26bd
0458770
6a52e2b
f820c61
4372170
1debdfa
763706d
4de6ec1
9422a4f
52bdf48
1e95df3
d59b37b
6a7979d
fd87585
e566d79
3be78c4
e51de8f
b600089
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1047,7 +1047,10 @@ object ReplaceDistinctWithAggregate extends Rule[LogicalPlan] { | |
| * ==> SELECT DISTINCT a1, a2 FROM Tab1 LEFT SEMI JOIN Tab2 ON a1<=>b1 AND a2<=>b2 | ||
| * }}} | ||
| * | ||
| * This rule is only applicable to INTERSECT DISTINCT. Do not use it for INTERSECT ALL. | ||
| * Note: | ||
| * 1. This rule is only applicable to INTERSECT DISTINCT. Do not use it for INTERSECT ALL. | ||
| * 2. This rule has to be done after de-duplicating the attributes; otherwise, the generated | ||
| * join conditions will be incorrect. | ||
| */ | ||
| object ReplaceIntersectWithSemiJoin extends Rule[LogicalPlan] { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we need to add a comment at here to mention that this rewrite is just for
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, will do it. Actually, I will also implement |
||
| def apply(plan: LogicalPlan): LogicalPlan = plan transform { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. use transformUp? cc @yhuai
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. actually nvm. |
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -316,8 +316,6 @@ abstract class BinaryNode extends LogicalPlan { | |
|
|
||
| override def children: Seq[LogicalPlan] = Seq(left, right) | ||
|
|
||
| def duplicateResolved: Boolean = left.outputSet.intersect(right.outputSet).isEmpty | ||
|
|
||
| override lazy val resolved: Boolean = | ||
| expressions.forall(_.resolved) && childrenResolved && duplicateResolved | ||
| expressions.forall(_.resolved) && childrenResolved | ||
|
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here, we could say
Intersectdirectly