Skip to content

[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation #8038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 14 commits into from
Closed

[SPARK-9747] [SQL] Avoid starving an unsafe operator in aggregation #8038

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
27f2e7f
Reserve memory in advance in TungstenAggregate
Aug 7, 2015
6549654
Actually request the memory in constructor + add tests
Aug 7, 2015
995be3d
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 7, 2015
ca1b44c
Minor: Update comment
Aug 7, 2015
355a9bd
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 7, 2015
2cf088a
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 10, 2015
4d416d0
Do not acquire a page if creating sorter destructively
Aug 10, 2015
70b8500
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 10, 2015
b4d3633
Address comments
Aug 10, 2015
b10a4f3
Fix test
Aug 11, 2015
94ca5de
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 11, 2015
d4dc9ca
Fix tests
Aug 12, 2015
19f2e1b
Fix tests again
Aug 12, 2015
7ebf6b9
Merge branch 'master' of github.com:apache/spark into unsafe-starve-m…
Aug 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix tests
  • Loading branch information
Andrew Or committed Aug 12, 2015
commit d4dc9cae8608622f25948aac51a29953cd25b09f
6 changes: 4 additions & 2 deletions ...est/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIteratorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package org.apache.spark.sql.execution.aggregate
import org.apache.spark._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection
import org.apache.spark.sql.execution.metric.LongSQLMetric
import org.apache.spark.sql.test.TestSQLContext
import org.apache.spark.unsafe.memory.TaskMemoryManager

Expand All @@ -39,8 +40,9 @@ class TungstenAggregationIteratorSuite extends SparkFunSuite {
val newMutableProjection = (expr: Seq[Expression], schema: Seq[Attribute]) => {
() => new InterpretedMutableProjection(expr, schema)
}
iter = new TungstenAggregationIterator(
Seq.empty, Seq.empty, Seq.empty, 0, Seq.empty, newMutableProjection, Seq.empty, None)
val dummyAccum = new LongSQLMetric("dummy")
iter = new TungstenAggregationIterator(Seq.empty, Seq.empty, Seq.empty, 0,
Seq.empty, newMutableProjection, Seq.empty, None, dummyAccum, dummyAccum)
val numPages = iter.getHashMap.getNumDataPages
assert(numPages === 1)
} finally {
Expand Down