Skip to content

[SPARK-20636] Add new optimization rule to transpose adjacent Window expressions. #17899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ptkool wants to merge 12 commits into from
Closed

[SPARK-20636] Add new optimization rule to transpose adjacent Window expressions. #17899

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f444b2c
Add new optimization rule to flip adjacent Window expressions.
ptkool May 8, 2017
6f8c036
Changes based on feedback.
ptkool May 9, 2017
b7793cd
Resolve Scala style errors.
ptkool May 9, 2017
f183b78
Resolve more Scala style issues.
ptkool May 9, 2017
1f21841
Small tweak to rule.
ptkool May 9, 2017
9a28535
Changes based on feedback.
ptkool May 20, 2017
76f1721
New test case.
ptkool Jul 12, 2017
f840c69
Fix scalastyle errors
ptkool Oct 29, 2017
e9f6928
Fix another scalastyle error
ptkool Oct 29, 2017
72a4b3a
Resolve conflicts
ptkool Jan 18, 2018
94e8115
Resolve conflicts
ptkool Sep 6, 2018
4328831
Remove style changes
ptkool Sep 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Remove style changes
  • Loading branch information
@ptkool
ptkool committed Sep 7, 2018
commit 432883117b8d067a821422d68fcfc3a03ca36527
18 changes: 9 additions & 9 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -726,10 +726,10 @@ object CollapseRepartition extends Rule[LogicalPlan] {
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
// This assumes Window contains the same type of window expressions. This is ensured
// by ExtractWindowFunctions.
WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
// This assumes Window contains the same type of window expressions. This is ensured
// by ExtractWindowFunctions.
WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
}
}
Expand All @@ -748,11 +748,11 @@ object TransposeWindow extends Rule[LogicalPlan] {

def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
Copy link
Member

@gatorsmile gatorsmile Jun 30, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expressions in both w1.expressions and w2.expressions must be deterministic. If not, we should not flip

Copy link
Contributor Author

@ptkool ptkool Jun 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? This seems overly restrictive to me.

Copy link
Member

@gatorsmile gatorsmile Oct 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to ensure the results are still the same with and without the rule.

if w1.references.intersect(w2.windowOutputSet).isEmpty &&
w1.expressions.forall(_.deterministic) &&
w2.expressions.forall(_.deterministic) &&
compatibleParititions(ps1, ps2) =>
Project(w1.output, Window(we2, ps2, os2, Window(we1, ps1, os1, grandChild)))
if w1.references.intersect(w2.windowOutputSet).isEmpty &&
w1.expressions.forall(_.deterministic) &&
w2.expressions.forall(_.deterministic) &&
compatibleParititions(ps1, ps2) =>
Project(w1.output, Window(we2, ps2, os2, Window(we1, ps1, os1, grandChild)))
Copy link
Contributor

@hvanhovell hvanhovell May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably warrants a follow-up that tries to move projections that are wedged in between two window clauses.

}
}

Expand Down