Skip to content

[SPARK-20175][SQL] Exists should not be evaluated in Join operator #17491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 4 commits into from
Closed

[SPARK-20175][SQL] Exists should not be evaluated in Join operator #17491

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b012550
Exists should not be evaluated in Join operator too.
viirya Mar 31, 2017
88016bf
Add a local limit operator to limit data scan.
viirya Apr 1, 2017
329f067
Merge remote-tracking branch 'upstream/master' into dont-push-exists-…
viirya Apr 11, 2017
24ae5ce
Revert optimization for Exists subquery without correlated references.
viirya Apr 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'upstream/master' into dont-push-exists-… 
…to-join
  • Loading branch information
@viirya
viirya committed Apr 11, 2017
commit 329f067d1a38469675367f1e29330034f6d923e8
2 changes: 2 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ trait PredicateHelper {
* Returns true iff `expr` could be evaluated as a condition within join.
*/
protected def canEvaluateWithinJoin(expr: Expression): Boolean = expr match {
// Non-deterministic expressions are not allowed as join conditions.
case e if !e.deterministic => false
case _: ListQuery | _: Exists =>
// A ListQuery defines the query which we want to search in an IN subquery expression.
// Currently the only way to evaluate an IN subquery is to convert it to a
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.