Skip to content

[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions #10542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gatorsmile wants to merge 37 commits into from
Closed

[SPARK-12594] [SQL] Outer Join Elimination by Filter Conditions #10542

Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2f6c80d
Merge remote-tracking branch 'upstream/master' into outerJoinElimination
gatorsmile Dec 31, 2015
90576aa
outer join conversion
gatorsmile Jan 1, 2016
5adec63
[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.…
JoshRosen Jan 1, 2016
192ab19
added test cases.
gatorsmile Jan 1, 2016
c9dbfcc
[SPARK-11743][SQL] Move the test for arrayOfUDT
viirya Jan 1, 2016
a59a357
[SPARK-3873][MLLIB] Import order fixes.
Jan 1, 2016
ad5b7cf
[SPARK-12409][SPARK-12387][SPARK-12391][SQL] Refactor filter pushdown…
viirya Jan 1, 2016
c04b53b
renaming
gatorsmile Jan 1, 2016
01a2986
[SPARK-12592][SQL][TEST] Don't mute Spark loggers in TestHive.reset()
liancheng Jan 1, 2016
6c20b3c
Disable test-dependencies.sh.
rxin Jan 1, 2016
0da7bd5
[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][SQL] always outp…
Jan 1, 2016
44ee920
Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][SQL] alw…
rxin Jan 2, 2016
970635a
[SPARK-12362][SQL][WIP] Inline Hive Parser
hvanhovell Jan 2, 2016
94f7a12
[SPARK-10180][SQL] JDBC datasource are not processing EqualNullSafe f…
HyukjinKwon Jan 2, 2016
15bd736
[SPARK-12481][CORE][STREAMING][SQL] Remove usage of Hadoop deprecated…
srowen Jan 2, 2016
65f9125
extend the condition to cover more cases in non null predicates.
gatorsmile Jan 3, 2016
513e3b0
[SPARK-12599][MLLIB][SQL] Remove the use of callUDF in MLlib
rxin Jan 3, 2016
6c5bbd6
Revert "Revert "[SPARK-12286][SPARK-12290][SPARK-12294][SPARK-12284][…
rxin Jan 3, 2016
9398644
added three more expressions: and, or and not
gatorsmile Jan 3, 2016
0bb07cb
style fix.
gatorsmile Jan 3, 2016
c5ff632
support non-local predicates and bug fix.
gatorsmile Jan 3, 2016
c3d5056
[SPARK-12327][SPARKR] fix code for lintr warning for commented code
felixcheung Jan 3, 2016
ee29dd2
scala style fix.
gatorsmile Jan 3, 2016
c82924d
[SPARK-12533][SQL] hiveContext.table() throws the wrong exception
thomastechs Jan 3, 2016
7b92922
Update MimaExcludes now Spark 1.6 is in Maven.
rxin Jan 4, 2016
b8410ff
[SPARK-12537][SQL] Add option to accept quoting of all character back…
Cazen Jan 4, 2016
13dab9c
[SPARK-12611][SQL][PYSPARK][TESTS] Fix test_infer_schema_to_local
holdenk Jan 4, 2016
7b7ea90
outer join conversion
gatorsmile Jan 1, 2016
7558e70
added test cases.
gatorsmile Jan 1, 2016
d3cbf46
renaming
gatorsmile Jan 1, 2016
2535cb1
extend the condition to cover more cases in non null predicates.
gatorsmile Jan 3, 2016
6c3f4b0
added three more expressions: and, or and not
gatorsmile Jan 3, 2016
fcd757c
style fix.
gatorsmile Jan 3, 2016
5bc7f52
support non-local predicates and bug fix.
gatorsmile Jan 3, 2016
34a0056
scala style fix.
gatorsmile Jan 3, 2016
ee7db1a
code refactoring and code merge
gatorsmile Jan 4, 2016
63d5d62
Merge remote-tracking branch 'origin/outerJoinConversion' into outerJ…
gatorsmile Jan 4, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
extend the condition to cover more cases in non null predicates.
  • Loading branch information
@gatorsmile
gatorsmile committed Jan 4, 2016
commit 2535cb1f3ecedda230734da7f644edfaaf03dd6c
26 changes: 13 additions & 13 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -787,19 +787,19 @@ object OuterJoinElimination extends Rule[LogicalPlan] with PredicateHelper {
private def hasNonNullPredicate(condition: Seq[Expression], child: LogicalPlan): Boolean = {
val localCondition = condition.filter(_.references subsetOf child.outputSet)
localCondition.exists(_.collect {
case EqualTo(ar: AttributeReference, l: Literal) => true
case EqualTo(l: Literal, ar: AttributeReference) => true
case EqualNullSafe(ar: AttributeReference, l: Literal) => true
case EqualNullSafe(l: Literal, ar: AttributeReference) => true
case GreaterThan(ar: AttributeReference, l: Literal) => true
case GreaterThan(l: Literal, ar: AttributeReference) => true
case GreaterThanOrEqual(ar: AttributeReference, l: Literal) => true
case GreaterThanOrEqual(l: Literal, ar: AttributeReference) => true
case LessThan(ar: AttributeReference, l: Literal) => true
case LessThan(l: Literal, ar: AttributeReference) => true
case LessThanOrEqual(ar: AttributeReference, l: Literal) => true
case LessThanOrEqual(l: Literal, ar: AttributeReference) => true
case In(ar: AttributeReference, l) => true
case EqualTo(ar: AttributeReference, _) => true
case EqualTo(_, ar: AttributeReference) => true
case EqualNullSafe(ar: AttributeReference, l) if !l.nullable => true
case EqualNullSafe(l, ar: AttributeReference) if !l.nullable => true
case GreaterThan(ar: AttributeReference, _) => true
case GreaterThan(_, ar: AttributeReference) => true
case GreaterThanOrEqual(ar: AttributeReference, _) => true
case GreaterThanOrEqual(_, ar: AttributeReference) => true
case LessThan(ar: AttributeReference, _) => true
case LessThan(_, ar: AttributeReference) => true
case LessThanOrEqual(ar: AttributeReference, _) => true
case LessThanOrEqual(_, ar: AttributeReference) => true
case In(ar: AttributeReference, _) => true
case IsNotNull(ar: AttributeReference) => true
}.nonEmpty)
}
Expand Down