-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-31705][SQL] Push more possible predicates through Join via CNF conversion #28733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
054594f
6591680
3497b3c
ada4135
7e4b019
0c92c1c
7853f67
84e89de
0af4c48
95ee45e
6bf4747
fa03b00
c225f74
296068c
377f9d8
be79ab7
af018be
b42ce1d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -227,30 +227,27 @@ trait PredicateHelper extends Logging { | |
| // For each side, there is no need to expand predicates of the same references. | ||
| // So here we can aggregate predicates of the same references as one single predicate, | ||
| // for reducing the size of pushed down predicates and corresponding codegen. | ||
| val right = aggregateExpressionsOfSameQualifiers(resultStack.pop()) | ||
| val left = aggregateExpressionsOfSameQualifiers(resultStack.pop()) | ||
| val right = groupExpressionsByQualifier(resultStack.pop()) | ||
| val left = groupExpressionsByQualifier(resultStack.pop()) | ||
| // Stop the loop whenever the result exceeds the `maxCnfNodeCount` | ||
| if (left.size * right.size > maxCnfNodeCount) { | ||
cloud-fan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Seq.empty | ||
| return Seq.empty | ||
| } else { | ||
| for {x <- left; y <- right} yield Or(x, y) | ||
|
||
| } | ||
| case other => other :: Nil | ||
| } | ||
| if (cnf.isEmpty) { | ||
| return Seq.empty | ||
| } | ||
| if (resultStack.length != 1) { | ||
| logWarning("The length of CNF conversion result stack is supposed to be 1. There might " + | ||
| "be something wrong with CNF conversion.") | ||
| return Seq.empty | ||
| } | ||
| resultStack.push(cnf) | ||
| } | ||
| if (resultStack.length != 1) { | ||
| logWarning("The length of CNF conversion result stack is supposed to be 1. There might " + | ||
| "be something wrong with CNF conversion.") | ||
| return Seq.empty | ||
| } | ||
| resultStack.top | ||
| } | ||
|
|
||
| private def aggregateExpressionsOfSameQualifiers( | ||
| private def groupExpressionsByQualifier( | ||
| expressions: Seq[Expression]): Seq[Expression] = { | ||
|
||
| expressions.groupBy(_.references.map(_.qualifier)).map(_._2.reduceLeft(And)).toSeq | ||
| } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For a test case See qualifier when groupby , they are
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can try I will update this PR later
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Not work, just
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The qualifier is the table name which is able to be used for aggregating more expressions
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Got the point, you did this for split condition to join children, I want convert scan predicate condition to optimize scan predicate.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this PR is complex enough. Let's keep this part in this way for now.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yea, I will raise pr for other problem base on your code and change a little after your pr merged. |
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the same references.->the same table references.?