Skip to content

[SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys #10896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 21 commits into from
Closed

[SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys #10896

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
85a845f
Skip unnecessary final group-by when input data already clustered
maropu Jan 25, 2016
f3659b7
Rename an argument in planAggregateWithoutDistinct
maropu Feb 5, 2016
c4242ea
Fix style errors
maropu Apr 25, 2016
a5addeb
Apply comments
maropu Jun 7, 2016
74fa277
Re-Implement in EnsureRequirements
maropu Jun 14, 2016
b747085
Fix bugs in planStreamingAggregation
maropu Jun 16, 2016
cb51727
Fix comments in AggUtils
maropu Jun 16, 2016
5de4871
Refactor logics in EnsureRequirements
maropu Jun 16, 2016
dc6a7f2
Fix bugs
maropu Jun 17, 2016
7aff3ad
Fix tests
maropu Jun 19, 2016
000c501
Update comments
maropu Jun 19, 2016
2615a6e
Add tests
maropu Jun 19, 2016
91701eb
Fix a syntax error
maropu Aug 19, 2016
74a14d7
Add tests
maropu Aug 22, 2016
e37ef6a
Add a superclass for *AggregateExec
maropu Aug 22, 2016
8b64305
Replace isSupportPartial with an extractor
maropu Aug 22, 2016
0375ac6
Rename the extractor
maropu Aug 22, 2016
86068d0
Replace trait with sueprclass
maropu Aug 22, 2016
69751bb
Apply minor comments
maropu Aug 23, 2016
d5e0ed3
Add a new function to check if it supports partial agg
maropu Aug 24, 2016
ac68145
Apply comments
maropu Aug 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix a syntax error
  • Loading branch information
@maropu
maropu committed Aug 19, 2016
commit 91701eb5171555ce73ba6b190b41cc17f05eee8c
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class PlannerSuite extends SharedSQLContext {
}

test("non-partial aggregation for distinct aggregates") {
Copy link
Contributor

@hvanhovell hvanhovell Aug 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add a test for non-partial aggregation with a regular (non distinct) aggregate?

Copy link
Member Author

@maropu maropu Aug 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

withTempTable("testNonPartialAggregation") {
withTempView("testNonPartialAggregation") {
val schema = StructType(StructField(s"value", IntegerType, true) :: Nil)
val row = Row.fromSeq(Seq.fill(1)(null))
val rowRDD = sparkContext.parallelize(row :: Nil)
Expand Down