You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The reason will be displayed to describe this comment to others. Learn more.
So I am not sure I agree with you here.
Lets take your example. We filter the rows with the following predicate: key + value > 2. We use both key and value in PARTITION BY clause, this means they are constant during window function evaluation. The value of key + value > 1 will (as a result) also be constant during window evaluation. This predicate would filter out entire partitions, so we can safely push it down.
I think we can safely push down any deterministic filter which only references partitioning columns. Let me know what you think.
I am not sure why Hive does not push this down, but this could well be due to the way Hive evaluates window functions (PTFs).
The reason will be displayed to describe this comment to others. Learn more.
You are right. In our window function evaluation, both key and value are constant. It should be safe to push down key + value > 2. Will follow your idea. Thanks!
// 2. Window partitioning key should be just a sequence of [[AttributeReference]].
The reason will be displayed to describe this comment to others. Learn more.
How about this case? It appears like key + value is also constant when evaluating window functions. It should be OK to push it down? This restriction could be also related to Hive implementation?
The reason will be displayed to describe this comment to others. Learn more.
@gatorsmile this restriction is probably more related to the fact that if we push down an entire expression, e.g.: a + b, we have to evaluate the expression twice, once in the Filter and once in the Window function. The double evaluation could be avoided by planning a Project.
I am pretty sure that we move all expressions used in Window clauses into an underlying Project during Analysis. So this shouldn't be to big of a problem.
The reason will be displayed to describe this comment to others. Learn more.
Yeah, thank you very much! I just added an extra condition to ensure that Analyzer converts all the compound expressions to alias. Added a couple of test cases to ensure it.
To enable predicate push down when the partitioning columns is a + b and the predicate is a + b > 3, we need to add a rule in Analyzer for converting the expressions to the underlying alias that has the exactly same original expressions. This will be submitted in a separate PR.
// 3. deterministic
case filter@Filter(condition, w: Window)
case filter @ Filter(condition, w: Window)
if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
val partitionAttrs = AttributeSet(w.partitionSpec.flatMap(_.references))
val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition { cond =>
Expand Down
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can push any deterministic predicate down as all of its references are part of the partition key. So why limit this?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, we are unable to push the predicate
(key + value) > '2'in the following query. Predicate push down will change the value ofsum(key)The example is copied from the test case of Hive. https://issues.apache.org/jira/secure/attachment/12788757/HIVE-12808.05.patch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I am not sure I agree with you here.
Lets take your example. We filter the rows with the following predicate:
key + value > 2. We use bothkeyandvalueinPARTITION BYclause, this means they are constant during window function evaluation. The value ofkey + value > 1will (as a result) also be constant during window evaluation. This predicate would filter out entire partitions, so we can safely push it down.I think we can safely push down any deterministic filter which only references partitioning columns. Let me know what you think.
I am not sure why Hive does not push this down, but this could well be due to the way Hive evaluates window functions (PTFs).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right. In our window function evaluation, both
keyandvalueare constant. It should be safe to push downkey + value > 2. Will follow your idea. Thanks!