Skip to content

[SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in RewriteDistinctAggregates #37825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bersprockets wants to merge 13 commits into from
Closed

[SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in RewriteDistinctAggregates #37825

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a5a6fc0
First attempt
bersprockets Mar 11, 2022
0a109d9
Fix for foldables
bersprockets Mar 11, 2022
38f1f6a
Update
bersprockets Mar 18, 2022
3fa3588
Update test
bersprockets Mar 18, 2022
4a40f91
Simplify
bersprockets Sep 3, 2022
165f558
Rename
bersprockets Sep 4, 2022
27dcffe
Update comments
bersprockets Sep 7, 2022
484ca8e
Replace Symbol usage with $"" in new unit tests
bersprockets Sep 7, 2022
208fe82
Update tests
bersprockets Sep 8, 2022
882cdaa
Update
bersprockets Sep 26, 2022
f53136d
Use ExpressionSet as key for various distinct aggregate child maps
bersprockets Sep 27, 2022
9938252
Handle case of one distinct grouping with superficially different fun…
bersprockets Oct 3, 2022
f7d29df
Update
bersprockets Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Replace Symbol usage with $"" in new unit tests
  • Loading branch information
@bersprockets
bersprockets committed Oct 11, 2022
commit 484ca8ed206775b4e8bc7ea016c95d2e386d4a9c
20 changes: 10 additions & 10 deletions ...c/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregatesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ class RewriteDistinctAggregatesSuite extends PlanTest {

test("eliminate multiple distinct groups due to superficial differences") {
val input = testRelation
.groupBy('a)(
countDistinct('b + 'c).as('agg1),
countDistinct('c + 'b).as('agg2),
max('c).as('agg3))
.groupBy($"a")(
countDistinct($"b" + $"c").as("agg1"),
countDistinct($"c" + $"b").as("agg2"),
max($"c").as("agg3"))
.analyze

val rewrite = RewriteDistinctAggregates(input)
Expand All @@ -93,12 +93,12 @@ class RewriteDistinctAggregatesSuite extends PlanTest {

test("reduce multiple distinct groups due to superficial differences") {
val input = testRelation
.groupBy('a)(
countDistinct('b + 'c + 'd).as('agg1),
countDistinct('d + 'c + 'b).as('agg2),
countDistinct('b + 'c).as('agg4),
countDistinct('c + 'b).as('agg4),
max('c).as('agg5))
.groupBy($"a")(
countDistinct($"b" + $"c" + $"d").as("agg1"),
countDistinct($"d" + $"c" + $"b").as("agg2"),
countDistinct($"b" + $"c").as("agg3"),
countDistinct($"c" + $"b").as("agg4"),
max($"c").as("agg5"))
.analyze

val rewrite = RewriteDistinctAggregates(input)
Expand Down