Skip to content

[SPARK-20636] Add new optimization rule to transpose adjacent Window expressions. #17899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ptkool wants to merge 12 commits into from
Closed

[SPARK-20636] Add new optimization rule to transpose adjacent Window expressions. #17899

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f444b2c
Add new optimization rule to flip adjacent Window expressions.
ptkool May 8, 2017
6f8c036
Changes based on feedback.
ptkool May 9, 2017
b7793cd
Resolve Scala style errors.
ptkool May 9, 2017
f183b78
Resolve more Scala style issues.
ptkool May 9, 2017
1f21841
Small tweak to rule.
ptkool May 9, 2017
9a28535
Changes based on feedback.
ptkool May 20, 2017
76f1721
New test case.
ptkool Jul 12, 2017
f840c69
Fix scalastyle errors
ptkool Oct 29, 2017
e9f6928
Fix another scalastyle error
ptkool Oct 29, 2017
72a4b3a
Resolve conflicts
ptkool Jan 18, 2018
94e8115
Resolve conflicts
ptkool Sep 6, 2018
4328831
Remove style changes
ptkool Sep 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Resolve Scala style errors.
  • Loading branch information
@ptkool
ptkool committed Oct 29, 2017
commit b7793cdf28283ff91bbcba9221a851930eec87b7
11 changes: 7 additions & 4 deletions ...atalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/TransposeWindowSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,16 +47,19 @@ class TransposeWindowSuite extends PlanTest {
val orderSpec2 = Seq(d.desc)

test("flip two adjacent windows with compatible partitions in multiple selects") {
val wexpr1 = windowExpr(sum('c), windowSpec(partitionSpec2, Seq.empty, UnspecifiedFrame))
val wexpr2 = windowExpr(sum('c), windowSpec(partitionSpec1, Seq.empty, UnspecifiedFrame))

val query = testRelation
.select('a, 'b, 'c, windowExpr(sum('c), windowSpec(partitionSpec2, Seq.empty, UnspecifiedFrame)).as('sum_a_2))
.select('a, 'b, 'c, 'sum_a_2, windowExpr(sum('c), windowSpec(partitionSpec1, Seq.empty, UnspecifiedFrame)).as('sum_a_1))
.select('a, 'b, 'c, wexpr1.as('sum_a_2))
.select('a, 'b, 'c, 'sum_a_2, wexpr2.as('sum_a_1))

val optimized = Optimize.execute(query.analyze)

val query2 = testRelation
.select('a, 'b, 'c)
.select('a, 'b, 'c, windowExpr(sum('c), windowSpec(partitionSpec1, Seq.empty, UnspecifiedFrame)).as('sum_a_1))
.select('a, 'b, 'c, windowExpr(sum('c), windowSpec(partitionSpec2, Seq.empty, UnspecifiedFrame)).as('sum_a_2), 'sum_a_1)
.select('a, 'b, 'c, wexpr2.as('sum_a_1))
.select('a, 'b, 'c, wexpr1.as('sum_a_2), 'sum_a_1)
.select('a, 'b, 'c, 'sum_a_2, 'sum_a_1)

val correctAnswer = Optimize.execute(query2.analyze)
Expand Down